2024-03-25T18:22:02.502 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:22:02.502 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324 --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/orig.config.yaml 2024-03-25T18:22:02.542 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:22:02.627 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621324' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIb/u/8YPBEaSOyeWaG1lD14x3+nk7rRSq7+W3fk8olAzDvJQ2CW4ABBgm3sDGoDCeheiNDR6yrJ/Zpl13Ne74E= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1vVUhWlSt6/30FLLdt0/GWS0KE8ZSYrh3OW3eK3B4lA2hSUSqcIhGCfPLa8vPgV7QJ24N99OVrmjZrrp019GA= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:22:02.627 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:22:02.627 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:22:02.628 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:22:02.629 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:22:02.651 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:22:02.651 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:22:02.651 INFO:teuthology.packaging:ref: None 2024-03-25T18:22:02.651 INFO:teuthology.packaging:tag: None 2024-03-25T18:22:02.651 INFO:teuthology.packaging:branch: main 2024-03-25T18:22:02.651 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:22:02.651 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:22:02.969 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:22:02.971 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:22:02.980 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:22:02.980 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:22:03.028 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:22:03.043 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:22:03.056 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:22:03.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:11.289146', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIb/u/8YPBEaSOyeWaG1lD14x3+nk7rRSq7+W3fk8olAzDvJQ2CW4ABBgm3sDGoDCeheiNDR6yrJ/Zpl13Ne74E='} 2024-03-25T18:22:03.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:11.290883', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1vVUhWlSt6/30FLLdt0/GWS0KE8ZSYrh3OW3eK3B4lA2hSUSqcIhGCfPLa8vPgV7QJ24N99OVrmjZrrp019GA='} 2024-03-25T18:22:03.105 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:22:03.116 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:22:03.116 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:22:03.116 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:22:03.250 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f58187340d0>, signals=[15]) 2024-03-25T18:22:03.250 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:22:03.261 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:22:03.261 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T18:22:03.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:03.351 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T18:22:03.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:03.437 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:22:03.446 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-03-25T18:22:03.463 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-03-25T18:22:03.464 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-03-25T18:22:03.520 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-03-25T18:22:03.520 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-03-25T18:22:03.520 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-03-25T18:22:03.520 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:22:03.520 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-03-25T18:22:03.520 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:22:03.521 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:22:03.521 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-03-25T18:22:03.521 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:22:03.521 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:22:03.521 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:22:03.521 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:22:03.521 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:22:03.521 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:22:03.521 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-03-25T18:22:03.561 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-03-25T18:22:03.578 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-03-25T18:22:03.579 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:22:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:22:03.636 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:22:03.636 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:22:03.636 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-03-25T18:22:03.668 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:22:03.683 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:22:03.716 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:22:03.716 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:22:03.718 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:22:03.734 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:22:03.745 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:22:03.745 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:22:03.775 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:22:03.836 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:22:03.896 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:22:03.910 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:22:03.910 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:22:03.911 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:22:03.911 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:22:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:03.927 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-03-25T18:22:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:03.928 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-25T18:22:03.985 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:22:03.985 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2024-03-25T18:22:03.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2024-03-25T18:22:03.986 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:22:03.986 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-361.el9.x86_64 2024-03-25T18:22:03.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-03-25T18:22:06.009 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T18:22:06.186 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T18:22:07.820 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-25T18:22:07.934 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 2.4 MB/s | 2.0 MB 00:00 2024-03-25T18:22:11.632 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-03-25T18:22:12.146 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:22:16.260 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 19 MB 00:03 2024-03-25T18:22:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-25T18:22:23.778 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:03 2024-03-25T18:22:24.844 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-25T18:22:25.267 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-25T18:22:26.323 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-25T18:22:26.369 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-25T18:22:27.350 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-25T18:22:31.477 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T18:22:32.300 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T18:22:33.609 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:22:33.609 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:22:33.706 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:22:33.717 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:22:33.717 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-25T18:22:33.717 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:22:33.717 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:22:33.717 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:22:33.717 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:22:33.717 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 76 M 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 123 M 2024-03-25T18:22:33.718 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:22:33.912 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:33.914 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:33.914 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:34.053 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:34.061 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:34.085 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:34.088 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:34.096 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:34.106 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.113 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.163 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:34.171 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.181 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.181 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.273 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.341 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:34.349 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:34.429 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:34.466 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:34.473 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:34.481 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:22:34.481 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:22:34.518 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:34.560 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:34.565 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:34.620 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:22:34.632 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:22:34.633 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:22:34.633 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:22:34.633 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2024-03-25T18:22:34.633 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:22:34.633 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 76 M 2024-03-25T18:22:34.633 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 123 M 2024-03-25T18:22:34.633 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:22:34.653 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:34.746 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:34.799 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:34.862 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:34.912 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.914 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.972 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:34.989 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:34.990 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.010 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:35.043 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.061 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.113 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:35.115 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:35.196 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:35.301 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:35.380 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:35.455 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:35.491 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:35.517 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:35.525 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.543 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.543 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.548 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:35.594 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:35.608 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.616 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.633 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:35.668 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:35.761 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:35.761 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 35 kB/s | 66 kB 00:01 2024-03-25T18:22:35.861 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:35.981 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:36.099 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:36.115 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:36.151 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:36.159 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:36.216 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:36.217 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:36.252 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:36.269 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:36.312 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:36.339 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:36.340 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:36.357 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:36.365 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:36.439 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:36.453 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:36.466 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:36.531 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:36.543 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:36.543 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:36.549 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:36.561 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:36.597 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:36.631 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:36.643 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:36.658 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:36.715 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:36.733 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:36.739 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:36.826 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:36.862 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 5.0 MB/s | 15 MB 00:03 2024-03-25T18:22:36.877 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:36.895 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:37.012 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:37.065 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:37.092 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 30 kB/s | 66 kB 00:02 2024-03-25T18:22:37.146 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:37.146 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:37.163 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:37.204 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:37.249 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:37.301 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:37.353 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:37.453 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:37.556 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:37.759 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:37.776 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:37.828 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:38.009 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.8 MB/s | 15 MB 00:03 2024-03-25T18:22:38.061 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:38.695 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.9 MB/s | 33 MB 00:04 2024-03-25T18:22:38.735 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 8.7 MB/s | 33 MB 00:03 2024-03-25T18:22:39.145 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 8.1 MB/s | 27 MB 00:03 2024-03-25T18:22:39.150 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:39.150 INFO:teuthology.orchestra.run.smithi169.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-25T18:22:39.486 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 11 MB/s | 27 MB 00:02 2024-03-25T18:22:39.490 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:39.490 INFO:teuthology.orchestra.run.smithi069.stdout:Total 16 MB/s | 76 MB 00:04 2024-03-25T18:22:39.814 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:22:40.155 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:22:40.211 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:22:40.211 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:22:40.531 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:22:40.531 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:22:41.259 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:22:41.259 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:22:41.539 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:22:41.540 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:22:43.739 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:22:44.007 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:22:44.338 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:44.429 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:44.674 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:44.768 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:47.222 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:47.358 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:47.538 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:47.677 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:52.200 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:52.347 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:52.385 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:57.105 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:26.398 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:26.398 INFO:teuthology.orchestra.run.smithi069.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:26.399 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:23:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:26.420 INFO:teuthology.orchestra.run.smithi169.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:23:26.443 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:26.836 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:26.836 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:26.836 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:26.836 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:26.875 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:26.876 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:26.876 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:26.876 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:27.229 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:27.229 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:23:27.229 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:23:27.229 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.229 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.229 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.229 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.229 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:23:27.230 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:23:27.277 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:23:27.577 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:27.625 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:27.679 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.680 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:27.680 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:27.681 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:27.681 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:27.681 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:23:27.681 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:27.681 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:27.760 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.760 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:27.760 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-03-25T18:23:27.771 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.772 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:27.772 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:27.772 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:27.772 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:27.772 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:23:27.772 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:27.772 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:27.798 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-03-25T18:23:27.798 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-03-25T18:23:27.798 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-03-25T18:23:27.798 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-03-25T18:23:27.798 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-03-25T18:23:27.798 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:27.799 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:27.800 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:27.800 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-03-25T18:23:27.800 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:23:27.800 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:27.800 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:27.801 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:27.801 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:27.852 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:27.852 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:27.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 68.el9 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8264254 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-03-25T18:23:27.895 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:23:27.896 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:27.897 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:27.897 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:27.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:28.496 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-03-25T18:23:28.522 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-03-25T18:23:29.876 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:29.888 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:29.915 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-03-25T18:23:29.918 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-03-25T18:23:29.922 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-03-25T18:23:29.924 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-25T18:23:29.933 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.jbE3AbEQXZ 2024-03-25T18:23:29.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jbE3AbEQXZ 2024-03-25T18:23:29.960 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.KtdgtWdHyL 2024-03-25T18:23:29.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KtdgtWdHyL 2024-03-25T18:23:30.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.jbE3AbEQXZ 2024-03-25T18:23:30.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.KtdgtWdHyL 2024-03-25T18:23:30.146 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.jbE3AbEQXZ is 6KB 2024-03-25T18:23:30.163 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.KtdgtWdHyL is 6KB 2024-03-25T18:23:30.193 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.jbE3AbEQXZ 2024-03-25T18:23:30.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:30.213 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.KtdgtWdHyL 2024-03-25T18:23:30.227 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:30.280 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:30.280 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:30.280 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:30.280 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:30.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:30.292 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:30.292 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:30.292 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:30.292 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:30.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:30.435 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-03-25T18:23:30.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-03-25T18:24:00.449 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:24:00.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-25T18:24:00.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:00.459 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:24:00.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T18:24:00.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:18.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-03-25T18:24:18.975 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-25T18:24:27.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-25T18:24:27.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:27.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T18:24:27.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:31.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-03-25T18:24:31.039 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-25T18:24:43.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-25T18:24:43.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:43.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T18:24:43.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:01.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-25T18:25:14.892 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-03-25T18:25:15.317 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-25T18:25:15.317 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:25:15.317 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-03-25T18:25:15.331 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:25:15.331 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:25:15.332 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:25:15.332 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:25:16.332 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:25:16.333 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:25:16.404 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-03-25T18:25:16.437 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:16.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T18:25:16.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:16.791 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-25T18:25:17.246 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T18:25:17.246 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:25:17.247 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-25T18:25:17.261 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:25:17.261 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:25:17.261 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:25:17.262 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:25:18.262 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:25:18.263 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:25:18.326 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-03-25T18:25:18.367 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:18.368 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:25:18.375 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:25:18.375 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:25:18.378 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:25:18.392 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:25:18.419 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:25:18.467 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:25:18.467 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:25:18.470 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:25:18.494 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:25:18.517 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:25:18.517 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:25:18.529 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:25:18.558 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:18.571 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:18.572 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:18.585 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:18.587 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:25:18.593 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:25:18.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:25:18.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:25:18.641 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:25:18.649 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:25:18.650 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:25:18.660 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:25:18.695 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:18.753 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:18.838 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:18.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:18.919 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T18:25:18.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:25:18.983 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:19.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:19.070 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:19.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:19.196 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T18:25:19.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:25:19.263 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-03-25T18:25:19.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-03-25T18:25:19.312 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:25:19.349 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:25:19.798 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:25:19.806 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:25:19.806 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:25:19.854 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:25:19.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-03-25T18:25:19.942 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-03-25T18:25:19.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-03-25T18:25:20.031 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-03-25T18:25:20.031 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:25:20.032 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-03-25T18:25:20.072 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-03-25T18:25:20.072 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-03-25T18:25:20.091 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-03-25T18:25:20.091 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:25:20.091 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:25:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:25:20.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:25:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:25:20.168 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:25:20.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:25:20.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:25:20.251 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:25:20.261 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:25:20.278 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:25:20.280 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:29:36.036 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-03-25T18:29:36.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-25T18:29:36.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:29:36.131 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-03-25T18:29:36.221 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-25T18:29:36.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T18:29:36.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:29:36.310 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-25T18:29:36.357 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T18:29:36.357 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:29:36.375 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:29:36.375 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:29:36.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:29:36.379 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:29:36.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:29:36.408 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:29:36.424 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:29:36.430 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:29:36.446 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:29:36.453 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-03-25T18:29:36.467 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-03-25T18:29:36.476 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-03-25T18:29:36.483 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:29:36.490 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-03-25T18:29:36.499 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:29:36.507 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:29:36.523 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:29:36.549 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:29:36.553 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:29:36.553 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-03-25T18:29:36.553 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:36.553 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:36.553 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:36.553 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:36.572 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:29:36.575 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:29:36.575 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-25T18:29:36.575 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:36.576 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:36.576 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:36.576 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:36.576 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:29:36.585 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:29:36.585 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:29:36.585 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:29:36.585 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:29:36.592 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:29:36.592 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:29:36.592 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:29:36.593 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:29:36.595 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:29:36.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:29:36.744 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:29:36.744 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:29:36.748 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:29:36.748 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:29:37.074 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:29:37.074 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T18:29:37.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:29:37.081 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:29:37.081 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T18:29:37.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:29:37.103 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:29:37.103 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:29:37.110 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:29:37.110 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:29:37.173 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:29:37.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:29:37.254 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:29:37.259 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:29:37.321 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-03-25T18:29:37.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-03-25T18:29:37.325 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-03-25T18:29:37.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-03-25T18:29:37.581 INFO:teuthology.orchestra.run.smithi169.stdout:74 files removed 2024-03-25T18:29:37.583 INFO:teuthology.orchestra.run.smithi069.stdout:74 files removed 2024-03-25T18:29:37.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-03-25T18:29:37.609 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw 2024-03-25T18:29:38.985 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 76 kB/s | 82 kB 00:01 2024-03-25T18:29:39.044 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 72 kB/s | 82 kB 00:01 2024-03-25T18:29:39.769 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 15 kB/s | 10 kB 00:00 2024-03-25T18:29:39.878 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:29:40.629 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T18:29:40.745 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T18:29:41.481 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.7 MB 00:00 2024-03-25T18:29:41.639 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 7.8 MB/s | 6.7 MB 00:00 2024-03-25T18:29:43.315 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-03-25T18:29:43.356 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-25T18:29:47.136 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-25T18:29:47.211 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-25T18:29:51.872 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 19 MB 00:03 2024-03-25T18:29:52.179 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:03 2024-03-25T18:29:58.909 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:29:59.600 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:03 2024-03-25T18:30:00.506 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-25T18:30:01.296 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-25T18:30:02.683 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T18:30:07.655 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T18:30:10.106 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:30:10.135 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:30:10.136 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:30:10.136 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:30:10.136 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:30:10.136 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:30:10.136 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-03-25T18:30:10.137 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:30:10.137 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:30:10.137 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:30:10.137 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options 2024-03-25T18:30:10.137 INFO:teuthology.orchestra.run.smithi069.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:30:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:30:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:30:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:30:10.142 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:30:10.142 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:30:10.142 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:30:10.142 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:30:10.142 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:30:10.142 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:30:10.143 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:30:10.143 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:30:10.143 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:30:10.143 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:30:10.143 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:30:10.143 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:30:10.143 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:30:10.143 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse 2024-03-25T18:30:10.144 INFO:teuthology.orchestra.run.smithi069.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:30:10.144 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:30:10.144 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:30:10.144 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:30:10.144 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:30:10.144 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:30:10.144 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:30:10.145 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:30:10.145 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:30:10.145 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:30:10.145 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:30:10.145 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2024-03-25T18:30:10.145 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:30:10.145 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:10.146 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:30:10.146 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:30:10.146 INFO:teuthology.orchestra.run.smithi069.stdout:Install 30 Packages 2024-03-25T18:30:10.146 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 2 Packages 2024-03-25T18:30:10.146 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:10.147 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 58 M 2024-03-25T18:30:10.147 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:30:11.681 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 1.9 MB/s | 21 MB 00:11 2024-03-25T18:30:16.614 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-25T18:30:17.522 INFO:teuthology.orchestra.run.smithi069.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 796 kB/s | 5.3 MB 00:06 2024-03-25T18:30:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 233 kB/s | 25 kB 00:00 2024-03-25T18:30:18.923 INFO:teuthology.orchestra.run.smithi069.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 574 kB/s | 742 kB 00:01 2024-03-25T18:30:19.067 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:30:19.083 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:30:19.083 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:30:19.084 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:30:19.084 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:30:19.084 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:30:19.084 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-03-25T18:30:19.084 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:30:19.084 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:30:19.084 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:30:19.084 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:30:19.086 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout:Install 30 Packages 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-03-25T18:30:19.087 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:30:19.088 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 58 M 2024-03-25T18:30:19.088 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:30:19.658 INFO:teuthology.orchestra.run.smithi069.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 678 kB/s | 498 kB 00:00 2024-03-25T18:30:23.473 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 5.3 MB 00:04 2024-03-25T18:30:23.573 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T18:30:24.107 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.4 MB/s | 742 kB 00:00 2024-03-25T18:30:24.465 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.4 MB/s | 498 kB 00:00 2024-03-25T18:30:25.133 INFO:teuthology.orchestra.run.smithi069.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 689 kB/s | 9.7 MB 00:14 2024-03-25T18:30:25.341 INFO:teuthology.orchestra.run.smithi069.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 217 kB/s | 45 kB 00:00 2024-03-25T18:30:25.658 INFO:teuthology.orchestra.run.smithi069.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 415 kB/s | 131 kB 00:00 2024-03-25T18:30:26.058 INFO:teuthology.orchestra.run.smithi069.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 405 kB/s | 162 kB 00:00 2024-03-25T18:30:26.358 INFO:teuthology.orchestra.run.smithi069.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 779 kB/s | 5.1 MB 00:06 2024-03-25T18:30:26.700 INFO:teuthology.orchestra.run.smithi069.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 872 kB/s | 298 kB 00:00 2024-03-25T18:30:26.751 INFO:teuthology.orchestra.run.smithi069.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 463 kB/s | 320 kB 00:00 2024-03-25T18:30:26.867 INFO:teuthology.orchestra.run.smithi069.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 908 kB/s | 106 kB 00:00 2024-03-25T18:30:26.893 INFO:teuthology.orchestra.run.smithi069.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 517 kB/s | 99 kB 00:00 2024-03-25T18:30:26.918 INFO:teuthology.orchestra.run.smithi069.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:30:26.935 INFO:teuthology.orchestra.run.smithi069.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T18:30:27.002 INFO:teuthology.orchestra.run.smithi069.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-25T18:30:27.027 INFO:teuthology.orchestra.run.smithi069.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-25T18:30:27.052 INFO:teuthology.orchestra.run.smithi069.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 831 kB/s | 41 kB 00:00 2024-03-25T18:30:27.087 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:27.087 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:27.167 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:27.167 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:27.305 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:27.323 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:27.372 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:27.390 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:27.497 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:27.521 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:27.659 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:27.683 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:28.016 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:28.034 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:28.043 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:28.086 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:28.088 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:28.156 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:28.164 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:28.206 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:28.208 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:28.242 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:28.378 INFO:teuthology.orchestra.run.smithi069.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 27 kB/s | 36 kB 00:01 2024-03-25T18:30:28.403 INFO:teuthology.orchestra.run.smithi069.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 160 kB/s | 221 kB 00:01 2024-03-25T18:30:28.421 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:28.527 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:28.749 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:28.845 INFO:teuthology.orchestra.run.smithi069.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 699 kB/s | 309 kB 00:00 2024-03-25T18:30:28.922 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:28.940 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 1.1 MB/s | 5.1 MB 00:04 2024-03-25T18:30:29.032 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 493 kB/s | 45 kB 00:00 2024-03-25T18:30:29.088 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:29.199 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 788 kB/s | 131 kB 00:00 2024-03-25T18:30:29.204 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:29.278 INFO:teuthology.orchestra.run.smithi069.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-25T18:30:29.294 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:29.311 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:29.366 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 971 kB/s | 162 kB 00:00 2024-03-25T18:30:29.370 INFO:teuthology.orchestra.run.smithi069.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-25T18:30:29.385 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:29.446 INFO:teuthology.orchestra.run.smithi069.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 657 kB/s | 49 kB 00:00 2024-03-25T18:30:29.456 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:29.472 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:29.512 INFO:teuthology.orchestra.run.smithi069.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T18:30:29.538 INFO:teuthology.orchestra.run.smithi069.stdout:(26/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 41 kB/s | 47 kB 00:01 2024-03-25T18:30:29.596 INFO:teuthology.orchestra.run.smithi069.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-25T18:30:29.741 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 855 kB/s | 320 kB 00:00 2024-03-25T18:30:29.755 INFO:teuthology.orchestra.run.smithi069.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-25T18:30:29.847 INFO:teuthology.orchestra.run.smithi069.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 620 kB/s | 192 kB 00:00 2024-03-25T18:30:30.709 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 883 kB/s | 9.7 MB 00:11 2024-03-25T18:30:30.792 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 284 kB/s | 298 kB 00:01 2024-03-25T18:30:30.950 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 411 kB/s | 99 kB 00:00 2024-03-25T18:30:30.976 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-03-25T18:30:31.018 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-25T18:30:31.043 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 660 kB/s | 44 kB 00:00 2024-03-25T18:30:31.093 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T18:30:31.118 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T18:30:31.144 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 831 kB/s | 41 kB 00:00 2024-03-25T18:30:31.208 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:31.210 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:31.336 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:31.353 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:31.360 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:31.397 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:31.431 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:31.449 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:31.599 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:31.618 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:31.949 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:31.975 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:32.017 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:32.040 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:32.147 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:32.170 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:32.183 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:32.218 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:32.252 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:32.271 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:32.272 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:32.289 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:32.429 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:32.454 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:32.520 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:32.528 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:32.636 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 24 kB/s | 36 kB 00:01 2024-03-25T18:30:32.661 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 143 kB/s | 221 kB 00:01 2024-03-25T18:30:32.701 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:32.826 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:32.842 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:32.895 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:33.028 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 842 kB/s | 309 kB 00:00 2024-03-25T18:30:33.053 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:33.303 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-25T18:30:33.370 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 398 kB/s | 26 kB 00:00 2024-03-25T18:30:33.437 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-03-25T18:30:33.465 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:33.504 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T18:30:33.542 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:33.588 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-25T18:30:33.589 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:33.605 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:33.688 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-25T18:30:33.698 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:33.718 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:33.788 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-25T18:30:33.899 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:33.975 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:34.014 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 34 kB/s | 47 kB 00:01 2024-03-25T18:30:35.005 INFO:teuthology.orchestra.run.smithi069.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 624 kB/s | 3.1 MB 00:05 2024-03-25T18:30:35.080 INFO:teuthology.orchestra.run.smithi069.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 649 kB/s | 3.4 MB 00:05 2024-03-25T18:30:38.914 INFO:teuthology.orchestra.run.smithi069.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 720 kB/s | 20 MB 00:28 2024-03-25T18:30:38.921 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:38.921 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.0 MB/s | 58 MB 00:28 2024-03-25T18:30:38.953 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:30:38.992 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:30:38.992 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:30:39.467 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:30:39.467 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:30:40.148 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 522 kB/s | 3.1 MB 00:06 2024-03-25T18:30:40.938 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:30:41.010 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:30:41.060 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:30:41.102 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:30:41.133 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:30:41.149 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:30:41.317 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:30:41.371 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:30:41.693 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:30:41.737 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:41.827 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:41.915 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:30:41.938 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:30:41.984 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:30:42.013 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:30:42.057 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:30:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:30:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:30:42.196 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:30:42.230 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:30:42.338 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:30:42.382 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:30:42.422 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:30:42.463 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:30:42.492 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:30:42.524 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:30:42.838 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:30:42.871 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:30:42.960 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:30:43.001 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:30:43.046 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:30:43.499 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:30:43.532 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:30:43.602 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:30:43.671 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:30:43.700 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:30:45.800 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:45.847 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:46.433 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:30:46.523 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:30:46.523 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:30:46.523 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:30:46.523 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:46.559 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:46.588 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 270 kB/s | 3.4 MB 00:12 2024-03-25T18:30:52.355 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 618 kB/s | 20 MB 00:32 2024-03-25T18:30:52.363 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:52.364 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.7 MB/s | 58 MB 00:33 2024-03-25T18:30:52.395 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:30:52.435 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:30:52.436 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:30:52.916 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:30:52.917 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:30:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:30:54.476 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:30:54.517 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:30:54.543 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:30:54.566 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:30:54.582 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:30:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:30:54.788 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:30:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:30:55.170 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:55.260 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:55.357 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:30:55.388 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:30:55.450 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:30:55.479 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:30:55.523 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:30:55.591 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:30:55.616 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:30:55.662 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:30:55.698 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-03-25T18:30:55.784 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:55.805 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:30:55.857 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:30:55.898 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:30:55.930 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:30:55.959 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:30:55.992 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:30:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:30:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:30:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:30:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:30:56.514 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:30:56.700 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:30:56.724 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:30:56.724 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:30:56.724 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:30:56.724 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:30:56.724 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:30:56.724 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:56.726 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:30:56.811 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:30:56.811 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:56.971 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:30:57.008 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:30:57.079 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:30:57.156 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:30:57.186 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:30:57.955 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:57.955 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:30:57.955 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:30:57.955 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:30:57.955 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:30:57.955 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:30:57.955 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:30:57.955 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:30:57.956 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:30:57.957 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:30:57.957 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:30:57.957 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:30:57.957 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:30:57.957 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:30:57.957 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:30:57.957 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:30:57.957 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:57.958 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:30:57.958 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.945 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:30:58.946 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:58.947 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:30:59.238 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test 2024-03-25T18:30:59.382 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:59.424 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:59.793 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:52 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:30:59.946 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:30:59.947 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:30:59.947 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:30:59.947 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:30:59.947 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:30:59.947 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:30:59.947 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:30:59.947 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:30:59.947 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:30:59.948 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 48 M 2024-03-25T18:30:59.949 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 204 M 2024-03-25T18:30:59.949 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:30:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:00.039 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:00.131 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:00.131 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:31:00.132 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:31:00.132 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:31:00.166 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:00.182 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:31:00.240 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T18:31:00.274 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T18:31:00.466 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 978 kB/s | 187 kB 00:00 2024-03-25T18:31:01.050 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 156 kB/s | 163 kB 00:01 2024-03-25T18:31:09.500 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:09.500 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-03-25T18:31:09.500 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-03-25T18:31:09.500 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-03-25T18:31:09.500 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-03-25T18:31:09.500 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-03-25T18:31:09.501 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-03-25T18:31:09.501 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-03-25T18:31:09.501 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:31:10.445 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:10.470 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:10.470 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:31:10.470 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:31:10.470 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:10.470 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:10.470 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:31:10.472 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:10.557 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:10.557 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:11.730 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:11.730 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:31:11.730 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:31:11.730 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:31:11.730 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:31:11.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:31:11.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:31:11.733 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:11.733 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:31:11.733 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.666 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:31:12.667 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:31:12.668 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:31:12.669 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:31:12.669 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:31:12.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-03-25T18:31:13.521 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:57 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:31:13.670 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:31:13.671 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:31:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-03-25T18:31:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:31:13.673 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 48 M 2024-03-25T18:31:13.673 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 204 M 2024-03-25T18:31:13.673 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:31:14.087 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T18:31:14.154 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.8 MB/s | 300 kB 00:00 2024-03-25T18:31:14.188 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T18:31:14.379 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-25T18:31:14.905 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 170 kB/s | 163 kB 00:00 2024-03-25T18:32:52.453 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 430 kB/s | 47 MB 01:52 2024-03-25T18:32:52.454 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:32:52.455 INFO:teuthology.orchestra.run.smithi069.stdout:Total 439 kB/s | 48 MB 01:52 2024-03-25T18:32:52.466 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:32:52.485 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:32:52.485 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:32:52.860 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:32:52.860 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:32:53.451 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:32:53.527 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:32:53.565 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:32:53.597 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:32:53.642 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:32:53.671 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:32:58.199 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:32:58.271 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:00.611 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:00.611 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:33:00.611 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:33:00.611 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:00.611 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:33:00.611 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:33:00.951 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:33:00.952 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:00.952 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:33:01.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph 2024-03-25T18:33:01.833 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:02:54 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:33:01.989 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:33:01.992 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2024-03-25T18:33:01.992 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:33:01.992 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2024-03-25T18:33:01.992 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:33:01.992 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:33:01.992 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:33:01.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:33:01.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:33:01.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:33:01.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:33:01.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:33:01.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:33:01.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:33:01.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:33:01.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:33:01.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:33:01.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:33:01.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:33:01.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:33:01.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:33:01.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:33:01.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:33:01.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:33:01.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:33:01.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:33:01.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:33:01.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:33:01.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:33:01.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout:Install 36 Packages 2024-03-25T18:33:01.997 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:01.998 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 30 M 2024-03-25T18:33:01.998 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 107 M 2024-03-25T18:33:01.999 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:33:02.637 INFO:teuthology.orchestra.run.smithi069.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T18:33:04.413 INFO:teuthology.orchestra.run.smithi069.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 709 kB/s | 1.5 MB 00:02 2024-03-25T18:33:06.104 INFO:teuthology.orchestra.run.smithi069.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 4.7 MB 00:03 2024-03-25T18:33:06.254 INFO:teuthology.orchestra.run.smithi069.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.6 MB/s | 247 kB 00:00 2024-03-25T18:33:06.371 INFO:teuthology.orchestra.run.smithi069.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:33:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:33:06.447 INFO:teuthology.orchestra.run.smithi069.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2024-03-25T18:33:06.481 INFO:teuthology.orchestra.run.smithi069.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:33:06.516 INFO:teuthology.orchestra.run.smithi069.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:33:06.558 INFO:teuthology.orchestra.run.smithi069.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:33:06.625 INFO:teuthology.orchestra.run.smithi069.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T18:33:06.650 INFO:teuthology.orchestra.run.smithi069.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 4.0 MB/s | 103 kB 00:00 2024-03-25T18:33:06.692 INFO:teuthology.orchestra.run.smithi069.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:33:06.726 INFO:teuthology.orchestra.run.smithi069.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T18:33:06.843 INFO:teuthology.orchestra.run.smithi069.stdout:(15/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86 474 kB/s | 2.1 MB 00:04 2024-03-25T18:33:06.910 INFO:teuthology.orchestra.run.smithi069.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 681 kB/s | 125 kB 00:00 2024-03-25T18:33:06.935 INFO:teuthology.orchestra.run.smithi069.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-25T18:33:07.027 INFO:teuthology.orchestra.run.smithi069.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-25T18:33:07.069 INFO:teuthology.orchestra.run.smithi069.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 951 kB/s | 151 kB 00:00 2024-03-25T18:33:07.128 INFO:teuthology.orchestra.run.smithi069.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T18:33:07.170 INFO:teuthology.orchestra.run.smithi069.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-25T18:33:07.211 INFO:teuthology.orchestra.run.smithi069.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 426 kB/s | 18 kB 00:00 2024-03-25T18:33:07.253 INFO:teuthology.orchestra.run.smithi069.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-03-25T18:33:07.279 INFO:teuthology.orchestra.run.smithi069.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-03-25T18:33:07.304 INFO:teuthology.orchestra.run.smithi069.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-25T18:33:07.329 INFO:teuthology.orchestra.run.smithi069.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-03-25T18:33:07.354 INFO:teuthology.orchestra.run.smithi069.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-03-25T18:33:07.380 INFO:teuthology.orchestra.run.smithi069.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-25T18:33:07.422 INFO:teuthology.orchestra.run.smithi069.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 396 kB/s | 16 kB 00:00 2024-03-25T18:33:07.472 INFO:teuthology.orchestra.run.smithi069.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-25T18:33:07.498 INFO:teuthology.orchestra.run.smithi069.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-25T18:33:07.523 INFO:teuthology.orchestra.run.smithi069.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-03-25T18:33:07.623 INFO:teuthology.orchestra.run.smithi069.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-25T18:33:07.648 INFO:teuthology.orchestra.run.smithi069.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-03-25T18:33:07.673 INFO:teuthology.orchestra.run.smithi069.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-03-25T18:33:16.315 INFO:teuthology.orchestra.run.smithi069.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.4 MB/s | 17 MB 00:11 2024-03-25T18:33:16.318 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:16.318 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.1 MB/s | 30 MB 00:14 2024-03-25T18:33:16.343 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:33:16.386 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:33:16.386 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:33:16.780 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:33:16.781 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:33:18.097 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:33:18.167 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:33:18.208 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:33:18.236 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:33:18.306 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:33:18.343 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:33:18.376 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:33:18.468 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:33:18.554 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:33:18.603 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:33:18.676 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:33:18.730 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:33:18.767 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:33:18.817 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:33:18.872 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:33:19.058 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:33:19.114 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:33:19.152 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:33:19.187 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:33:19.217 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:33:19.293 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:33:19.327 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:33:19.360 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:33:19.401 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:33:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:33:19.471 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:33:19.560 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:33:19.592 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:33:19.665 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:33:19.803 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:33:19.923 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:33:20.084 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:33:20.114 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:33:20.139 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:33:20.139 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:20.139 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:33:20.140 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:33:20.140 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:33:20.140 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:21.574 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:33:21.601 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:33:21.601 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:21.601 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:33:21.601 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:33:21.601 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:33:21.601 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:22.049 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:33:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:33:22.073 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:22.073 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:33:22.073 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:33:22.073 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:33:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:22.283 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:33:22.306 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:33:22.306 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:22.307 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:33:22.307 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:33:22.307 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:33:22.307 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:22.373 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:33:24.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:33:24.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:33:24.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:33:24.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:33:24.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:33:24.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:33:24.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:33:24.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:33:24.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:33:24.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:33:24.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:33:24.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:33:24.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:33:24.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:33:24.985 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:33:24.985 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:24.985 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:33:24.986 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:33:24.987 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:24.988 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:33:25.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base 2024-03-25T18:33:25.890 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:03:18 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:33:25.988 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:33:26.040 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:33:26.041 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:33:26.041 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:33:26.097 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install cephadm 2024-03-25T18:33:26.662 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:03:19 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:33:26.812 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:33:26.812 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:33:26.812 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:33:26.812 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:33:26.812 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:33:26.812 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:33:26.812 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:26.813 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:33:26.813 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:33:26.813 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:33:26.813 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:26.813 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 747 k 2024-03-25T18:33:26.813 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 754 k 2024-03-25T18:33:26.813 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:33:29.083 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 329 kB/s | 747 kB 00:02 2024-03-25T18:33:29.084 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:29.084 INFO:teuthology.orchestra.run.smithi069.stdout:Total 329 kB/s | 747 kB 00:02 2024-03-25T18:33:29.084 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:33:29.090 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:33:29.091 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:33:29.097 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:33:29.097 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:33:29.169 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:33:29.604 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:29.705 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:29.936 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 356 kB/s | 47 MB 02:15 2024-03-25T18:33:29.938 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:29.939 INFO:teuthology.orchestra.run.smithi169.stdout:Total 362 kB/s | 48 MB 02:16 2024-03-25T18:33:29.950 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:33:29.970 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:33:29.970 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:33:30.089 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:30.353 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:33:30.353 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:33:30.360 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:30.360 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:30.360 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:33:30.360 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:30.360 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:30.361 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:33:30.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:33:30.969 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:33:31.044 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:33:31.085 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:33:31.119 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:03:24 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:33:31.126 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:31.178 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:33:31.206 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:31.272 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:33:31.273 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:33:31.273 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T18:33:31.273 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:33:31.273 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 145 k 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 436 k 2024-03-25T18:33:31.274 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:33:31.939 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 219 kB/s | 145 kB 00:00 2024-03-25T18:33:31.940 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:31.940 INFO:teuthology.orchestra.run.smithi069.stdout:Total 218 kB/s | 145 kB 00:00 2024-03-25T18:33:31.940 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:33:31.948 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:33:31.948 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:33:32.002 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:33:32.002 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:33:32.140 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:33:32.198 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:32.223 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:32.223 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:32.223 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:33:32.224 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:33.265 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:33.265 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:33.265 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:33:33.265 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:33.265 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:33.265 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:33:33.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr 2024-03-25T18:33:34.058 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:03:27 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:33:34.154 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:33:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:33:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:33:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:33:34.259 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:33:34.815 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:03:27 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:33:34.964 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:33:34.965 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================== 2024-03-25T18:33:34.965 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================== 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================== 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout:Install 12 Packages 2024-03-25T18:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:34.968 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 6.9 M 2024-03-25T18:33:34.968 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 26 M 2024-03-25T18:33:34.968 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:33:35.627 INFO:teuthology.orchestra.run.smithi069.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-25T18:33:35.774 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:35.785 INFO:teuthology.orchestra.run.smithi069.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-25T18:33:35.827 INFO:teuthology.orchestra.run.smithi069.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T18:33:35.855 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:35.919 INFO:teuthology.orchestra.run.smithi069.stdout:(4/12): ceph-grafana-dashboards-19.0.0-1439.g5d 41 kB/s | 31 kB 00:00 2024-03-25T18:33:36.086 INFO:teuthology.orchestra.run.smithi069.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-03-25T18:33:36.328 INFO:teuthology.orchestra.run.smithi069.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T18:33:36.353 INFO:teuthology.orchestra.run.smithi069.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-25T18:33:36.554 INFO:teuthology.orchestra.run.smithi069.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 722 kB/s | 144 kB 00:00 2024-03-25T18:33:36.596 INFO:teuthology.orchestra.run.smithi069.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.6 MB/s | 2.0 MB 00:00 2024-03-25T18:33:36.629 INFO:teuthology.orchestra.run.smithi069.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 409 kB/s | 31 kB 00:00 2024-03-25T18:33:36.929 INFO:teuthology.orchestra.run.smithi069.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 565 kB/s | 188 kB 00:00 2024-03-25T18:33:38.345 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:38.346 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:33:38.346 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:33:38.346 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:38.346 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:33:38.346 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:33:38.735 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:33:38.736 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:33:38.736 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:33:39.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-03-25T18:33:39.563 INFO:teuthology.orchestra.run.smithi069.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 414 kB/s | 1.8 MB 00:04 2024-03-25T18:33:39.566 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:39.566 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.5 MB/s | 6.9 MB 00:04 2024-03-25T18:33:39.587 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:33:39.605 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:33:39.605 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:33:39.650 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:03:23 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:33:39.807 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:33:39.807 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:33:39.814 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:33:39.817 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-25T18:33:39.817 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:33:39.817 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-25T18:33:39.817 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:33:39.817 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:33:39.817 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:33:39.818 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:33:39.819 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout:Install 36 Packages 2024-03-25T18:33:39.820 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:33:39.821 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 30 M 2024-03-25T18:33:39.821 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 107 M 2024-03-25T18:33:39.821 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:33:40.317 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:33:40.430 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:33:40.431 INFO:teuthology.orchestra.run.smithi169.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T18:33:40.465 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:33:40.511 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:33:40.542 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:33:40.642 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:33:40.767 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:33:40.853 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:33:40.914 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:33:40.942 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:33:40.973 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:33:41.216 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:33:41.248 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:33:42.109 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:33:42.109 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:33:42.109 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:33:42.109 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:33:42.109 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:33:42.110 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:33:42.110 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:33:42.110 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:33:42.110 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:33:42.110 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:33:42.110 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:33:42.110 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:33:42.464 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:33:42.465 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:42.465 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:33:42.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:33:43.223 INFO:teuthology.orchestra.run.smithi169.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 468 kB/s | 1.5 MB 00:03 2024-03-25T18:33:43.264 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:03:36 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:33:43.414 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:33:43.416 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout:Install 12 Packages 2024-03-25T18:33:43.417 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:33:43.418 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 37 M 2024-03-25T18:33:43.418 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 209 M 2024-03-25T18:33:43.418 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:33:43.752 INFO:teuthology.orchestra.run.smithi069.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 213 kB/s | 32 kB 00:00 2024-03-25T18:33:43.794 INFO:teuthology.orchestra.run.smithi069.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-03-25T18:33:43.844 INFO:teuthology.orchestra.run.smithi069.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-03-25T18:33:43.995 INFO:teuthology.orchestra.run.smithi069.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.7 MB/s | 3.0 MB 00:00 2024-03-25T18:33:44.245 INFO:teuthology.orchestra.run.smithi069.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-03-25T18:33:44.306 INFO:teuthology.orchestra.run.smithi169.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 501 kB/s | 2.1 MB 00:04 2024-03-25T18:33:44.362 INFO:teuthology.orchestra.run.smithi069.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-25T18:33:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.5 MB/s | 217 kB 00:00 2024-03-25T18:33:44.665 INFO:teuthology.orchestra.run.smithi169.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 691 kB/s | 247 kB 00:00 2024-03-25T18:33:44.940 INFO:teuthology.orchestra.run.smithi169.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 680 kB/s | 187 kB 00:00 2024-03-25T18:33:44.996 INFO:teuthology.orchestra.run.smithi069.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-25T18:33:45.007 INFO:teuthology.orchestra.run.smithi169.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-25T18:33:45.029 INFO:teuthology.orchestra.run.smithi069.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-25T18:33:45.057 INFO:teuthology.orchestra.run.smithi169.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 634 kB/s | 32 kB 00:00 2024-03-25T18:33:45.096 INFO:teuthology.orchestra.run.smithi069.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-03-25T18:33:45.099 INFO:teuthology.orchestra.run.smithi169.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T18:33:45.141 INFO:teuthology.orchestra.run.smithi169.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 926 kB/s | 39 kB 00:00 2024-03-25T18:33:45.199 INFO:teuthology.orchestra.run.smithi169.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T18:33:45.222 INFO:teuthology.orchestra.run.smithi069.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 868 kB/s | 195 kB 00:00 2024-03-25T18:33:45.317 INFO:teuthology.orchestra.run.smithi169.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-03-25T18:33:45.367 INFO:teuthology.orchestra.run.smithi169.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-25T18:33:45.417 INFO:teuthology.orchestra.run.smithi169.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T18:33:45.468 INFO:teuthology.orchestra.run.smithi169.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T18:33:45.601 INFO:teuthology.orchestra.run.smithi169.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 937 kB/s | 125 kB 00:00 2024-03-25T18:33:45.643 INFO:teuthology.orchestra.run.smithi169.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 538 kB/s | 22 kB 00:00 2024-03-25T18:33:46.044 INFO:teuthology.orchestra.run.smithi169.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 378 kB/s | 151 kB 00:00 2024-03-25T18:33:46.160 INFO:teuthology.orchestra.run.smithi169.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 373 kB/s | 43 kB 00:00 2024-03-25T18:33:46.327 INFO:teuthology.orchestra.run.smithi169.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-25T18:33:46.586 INFO:teuthology.orchestra.run.smithi169.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-25T18:33:46.703 INFO:teuthology.orchestra.run.smithi169.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2024-03-25T18:33:46.795 INFO:teuthology.orchestra.run.smithi169.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T18:33:46.870 INFO:teuthology.orchestra.run.smithi169.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-03-25T18:33:46.970 INFO:teuthology.orchestra.run.smithi169.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-25T18:33:47.054 INFO:teuthology.orchestra.run.smithi169.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 236 kB/s | 20 kB 00:00 2024-03-25T18:33:47.154 INFO:teuthology.orchestra.run.smithi169.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T18:33:47.305 INFO:teuthology.orchestra.run.smithi169.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 526 kB/s | 79 kB 00:00 2024-03-25T18:33:47.372 INFO:teuthology.orchestra.run.smithi169.stdout:(28/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 691 kB/s | 4.7 MB 00:06 2024-03-25T18:33:47.413 INFO:teuthology.orchestra.run.smithi169.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-25T18:33:47.514 INFO:teuthology.orchestra.run.smithi169.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 899 kB/s | 90 kB 00:00 2024-03-25T18:33:47.573 INFO:teuthology.orchestra.run.smithi169.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 82 kB/s | 16 kB 00:00 2024-03-25T18:33:47.606 INFO:teuthology.orchestra.run.smithi169.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-03-25T18:33:47.740 INFO:teuthology.orchestra.run.smithi169.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-25T18:33:47.798 INFO:teuthology.orchestra.run.smithi169.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-03-25T18:33:47.823 INFO:teuthology.orchestra.run.smithi169.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-25T18:34:11.901 INFO:teuthology.orchestra.run.smithi069.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 267 kB/s | 7.4 MB 00:28 2024-03-25T18:34:11.903 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:11.904 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.3 MB/s | 37 MB 00:28 2024-03-25T18:34:11.947 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:34:11.973 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:34:11.973 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:34:12.331 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:34:12.331 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:34:13.056 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:34:13.124 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:34:13.164 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:34:13.256 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:13.300 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:34:13.476 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:13.626 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:34:14.262 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:34:14.386 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:34:15.610 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:34:16.648 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:34:16.682 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:19.169 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:19.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:34:19.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:34:19.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:19.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:34:19.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:19.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:19.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:34:19.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:34:19.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:34:19.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:34:19.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:34:19.623 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:34:19.625 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:34:19.625 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:34:19.625 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:19.625 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:19.980 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:34:20.353 INFO:teuthology.orchestra.run.smithi169.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 463 kB/s | 17 MB 00:37 2024-03-25T18:34:20.357 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:20.357 INFO:teuthology.orchestra.run.smithi169.stdout:Total 749 kB/s | 30 MB 00:40 2024-03-25T18:34:20.382 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:34:20.425 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:34:20.425 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:34:20.550 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:13 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:20.699 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:34:20.701 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout:Install 13 Packages 2024-03-25T18:34:20.702 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:20.703 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.0 M 2024-03-25T18:34:20.703 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 26 M 2024-03-25T18:34:20.703 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:34:20.839 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:34:20.839 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:34:20.982 INFO:teuthology.orchestra.run.smithi069.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 197 kB/s | 18 kB 00:00 2024-03-25T18:34:21.082 INFO:teuthology.orchestra.run.smithi069.stdout:(2/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T18:34:21.107 INFO:teuthology.orchestra.run.smithi069.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 112 kB/s | 24 kB 00:00 2024-03-25T18:34:21.132 INFO:teuthology.orchestra.run.smithi069.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-25T18:34:21.158 INFO:teuthology.orchestra.run.smithi069.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-25T18:34:21.183 INFO:teuthology.orchestra.run.smithi069.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-25T18:34:21.350 INFO:teuthology.orchestra.run.smithi069.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 90 kB/s | 15 kB 00:00 2024-03-25T18:34:21.375 INFO:teuthology.orchestra.run.smithi069.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-03-25T18:34:21.442 INFO:teuthology.orchestra.run.smithi069.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 95 kB/s | 52 kB 00:00 2024-03-25T18:34:21.501 INFO:teuthology.orchestra.run.smithi069.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-03-25T18:34:21.559 INFO:teuthology.orchestra.run.smithi069.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-25T18:34:21.618 INFO:teuthology.orchestra.run.smithi069.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-25T18:34:21.776 INFO:teuthology.orchestra.run.smithi069.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.5 MB/s | 1.0 MB 00:00 2024-03-25T18:34:21.778 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:21.779 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-25T18:34:21.779 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:34:21.792 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:34:21.792 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:34:21.880 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:34:21.882 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:34:22.159 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:34:22.230 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:34:22.281 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:34:22.325 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:34:22.393 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:34:22.394 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:34:22.440 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:34:22.473 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:34:22.563 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:34:22.564 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:34:22.610 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:34:22.634 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:34:22.674 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:34:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:34:22.711 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:34:22.753 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:34:22.764 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:34:22.818 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:34:22.846 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:34:22.856 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:34:22.905 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:34:22.956 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:34:22.960 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:34:23.005 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:34:23.144 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:34:23.219 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:34:23.249 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:34:23.284 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:34:23.314 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:34:23.390 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:34:23.424 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:23.458 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:34:23.498 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:34:23.541 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:34:23.585 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:34:23.675 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:34:23.716 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:34:23.735 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:34:23.777 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:34:23.789 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:34:23.810 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:34:23.859 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:34:23.889 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:34:23.958 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:34:24.085 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:24.243 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:34:24.272 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:24.297 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:24.297 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:24.297 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:34:24.297 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:24.297 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:24.297 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:24.733 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:34:24.733 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:34:24.733 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:34:24.733 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:34:24.733 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:34:24.733 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:34:24.734 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:34:24.734 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:34:24.734 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:34:24.734 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:34:24.734 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:34:24.734 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:34:24.734 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:34:25.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:34:25.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:34:25.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:34:25.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:34:25.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:34:25.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:34:25.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:34:25.094 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:25.094 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:25.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:34:25.807 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:25.834 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:34:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:25.835 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:25.911 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:26.061 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:26.061 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:34:26.061 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:34:26.063 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:26.063 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:34:26.063 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:34:26.063 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2024-03-25T18:34:26.063 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:26.064 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 6.8 M 2024-03-25T18:34:26.064 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 33 M 2024-03-25T18:34:26.064 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:34:26.282 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:26.305 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:26.306 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:26.306 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:34:26.306 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:26.306 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:26.306 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:26.453 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T18:34:26.525 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:26.548 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:26.548 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:26.548 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:34:26.548 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:26.548 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:26.548 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:26.614 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:26.720 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2024-03-25T18:34:26.745 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 436 kB/s | 248 kB 00:00 2024-03-25T18:34:26.779 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 983 kB/s | 58 kB 00:00 2024-03-25T18:34:26.862 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 201 kB/s | 138 kB 00:00 2024-03-25T18:34:26.896 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2024-03-25T18:34:26.897 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:26.897 INFO:teuthology.orchestra.run.smithi069.stdout:Total 8.1 MB/s | 6.8 MB 00:00 2024-03-25T18:34:26.910 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:34:26.920 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:34:26.920 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:34:27.008 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:34:27.008 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:34:27.332 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:34:27.501 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:34:27.555 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:34:28.073 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:34:28.150 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:34:28.210 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:34:28.234 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:34:28.421 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:34:28.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:34:28.423 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:34:28.423 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:28.423 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:34:28.423 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:34:28.423 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:34:28.423 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:34:28.423 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:34:29.059 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:34:29.059 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:34:29.059 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:34:29.059 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:34:29.059 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:34:29.059 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:29.364 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:34:29.409 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:34:29.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:29.411 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:34:29.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse 2024-03-25T18:34:29.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-03-25T18:34:30.149 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:30.298 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:30.298 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:30.298 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:34:30.298 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:30.298 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:34:30.298 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 964 k 2024-03-25T18:34:30.299 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.8 M 2024-03-25T18:34:30.300 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:34:30.331 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:34:30.414 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 690 kB/s | 79 kB 00:00 2024-03-25T18:34:30.429 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:30.481 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:34:30.482 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:34:30.482 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:34:30.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-03-25T18:34:31.111 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:34:31.264 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:34:31.264 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:34:31.264 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:34:31.264 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:34:31.264 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:34:31.265 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:34:31.265 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:31.265 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:34:31.265 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:34:31.265 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:34:31.265 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:31.266 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 747 k 2024-03-25T18:34:31.266 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 754 k 2024-03-25T18:34:31.266 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:34:32.272 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 449 kB/s | 885 kB 00:01 2024-03-25T18:34:32.274 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:32.275 INFO:teuthology.orchestra.run.smithi069.stdout:Total 488 kB/s | 964 kB 00:01 2024-03-25T18:34:32.275 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:34:32.298 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:34:32.298 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:34:32.376 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:34:32.376 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:34:32.556 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:34:32.698 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:34:32.763 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:34:33.466 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:34:33.467 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:34:33.648 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 314 kB/s | 747 kB 00:02 2024-03-25T18:34:33.649 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:33.649 INFO:teuthology.orchestra.run.smithi169.stdout:Total 313 kB/s | 747 kB 00:02 2024-03-25T18:34:33.649 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:34:33.656 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:34:33.656 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:34:33.662 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:34:33.662 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:34:33.741 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:34:33.783 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:34:33.783 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:33.783 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:34:33.784 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:34:33.784 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:33.784 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:34.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-volume 2024-03-25T18:34:34.219 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:34.311 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:34.581 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:34.730 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:34.730 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:34.730 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-25T18:34:34.730 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:34.730 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:34:34.730 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:34:34.731 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:34:34.731 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:34:34.731 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:34.731 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:34:34.731 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:34.731 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2024-03-25T18:34:34.731 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:34.732 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 332 k 2024-03-25T18:34:34.732 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.5 M 2024-03-25T18:34:34.732 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:34:34.877 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 475 kB/s | 69 kB 00:00 2024-03-25T18:34:34.986 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:34.986 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:34.986 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:34:34.986 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:34.986 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:34.986 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:34:35.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:34:35.710 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 269 kB/s | 264 kB 00:00 2024-03-25T18:34:35.712 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:35.712 INFO:teuthology.orchestra.run.smithi069.stdout:Total 339 kB/s | 332 kB 00:00 2024-03-25T18:34:35.712 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:34:35.721 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:34:35.721 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:34:35.759 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:34:35.784 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:34:35.906 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:34:35.943 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:34:35.943 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 145 k 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 436 k 2024-03-25T18:34:35.944 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:34:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:34:36.129 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:34:36.145 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:34:36.145 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:36.145 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:34:36.145 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:36.763 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 178 kB/s | 145 kB 00:00 2024-03-25T18:34:36.763 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:36.764 INFO:teuthology.orchestra.run.smithi169.stdout:Total 177 kB/s | 145 kB 00:00 2024-03-25T18:34:36.764 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:34:36.773 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:34:36.773 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:34:36.827 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:34:36.828 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:34:36.878 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:34:36.979 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:34:37.047 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:37.071 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:37.072 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:37.072 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:34:37.072 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:37.205 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:34:37.205 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:37.205 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:34:37.205 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:37.205 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:34:37.206 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:37.206 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:37.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel 2024-03-25T18:34:38.095 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:31 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:38.148 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:38.148 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:38.148 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:34:38.148 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:38.148 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:38.149 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:34:38.246 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:38.246 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 127 k 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 456 k 2024-03-25T18:34:38.247 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:34:38.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-03-25T18:34:38.985 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:22 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:34:38.991 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 170 kB/s | 127 kB 00:00 2024-03-25T18:34:38.991 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:38.992 INFO:teuthology.orchestra.run.smithi069.stdout:Total 170 kB/s | 127 kB 00:00 2024-03-25T18:34:38.992 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:34:38.997 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:34:38.997 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:34:39.054 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:34:39.054 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:34:39.083 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:39.134 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:34:39.135 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:34:39.135 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:34:39.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:34:39.204 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:34:39.314 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:39.730 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:39.760 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:34:39.910 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:34:39.911 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-25T18:34:39.911 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:34:39.911 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:34:39.912 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:34:39.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:34:39.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:34:39.913 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:39.913 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:34:39.913 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-25T18:34:39.913 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-03-25T18:34:39.913 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:39.914 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.9 M 2024-03-25T18:34:39.914 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-03-25T18:34:39.914 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:34:40.022 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:40.022 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:40.022 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:34:40.022 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:40.022 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:40.022 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:40.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2 2024-03-25T18:34:40.806 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:33 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:40.905 INFO:teuthology.orchestra.run.smithi069.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:40.959 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:40.960 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:34:40.960 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:41.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs-devel 2024-03-25T18:34:41.074 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 42 kB/s | 31 kB 00:00 2024-03-25T18:34:41.225 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T18:34:41.267 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T18:34:41.350 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 15 kB/s | 15 kB 00:01 2024-03-25T18:34:41.475 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-03-25T18:34:41.579 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:41.726 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-25T18:34:41.730 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:41.730 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:34:41.731 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:41.732 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 31 k 2024-03-25T18:34:41.732 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 140 k 2024-03-25T18:34:41.732 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:34:41.826 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T18:34:41.910 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-25T18:34:41.985 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 411 kB/s | 31 kB 00:00 2024-03-25T18:34:42.068 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T18:34:42.144 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.9 MB/s | 2.0 MB 00:00 2024-03-25T18:34:42.288 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 56 kB/s | 31 kB 00:00 2024-03-25T18:34:42.288 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:42.289 INFO:teuthology.orchestra.run.smithi069.stdout:Total 56 kB/s | 31 kB 00:00 2024-03-25T18:34:42.289 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:34:42.292 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:34:42.292 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:34:42.320 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:34:42.320 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:34:42.423 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:34:42.518 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:42.885 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:43.145 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:34:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:43.145 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:43.145 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:43.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2 2024-03-25T18:34:43.859 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:36 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:43.955 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:44.006 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:44.007 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:34:44.008 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:44.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1 2024-03-25T18:34:44.616 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:37 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:44.714 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:44.765 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:44.766 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:34:44.766 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:44.834 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados 2024-03-25T18:34:45.397 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:45.494 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:45.547 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:45.548 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:34:45.548 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:45.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw 2024-03-25T18:34:45.710 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 339 kB/s | 1.8 MB 00:05 2024-03-25T18:34:45.716 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:45.716 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.2 MB/s | 6.9 MB 00:05 2024-03-25T18:34:45.736 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:34:45.755 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:34:45.756 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:34:45.941 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:34:45.941 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:34:46.157 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:46.253 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:46.305 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:46.306 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:34:46.306 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:46.355 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs 2024-03-25T18:34:46.462 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:34:46.552 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:34:46.613 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:34:46.668 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:34:46.706 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:34:46.807 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:34:46.919 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:46.940 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:34:47.015 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:47.033 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:34:47.066 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:47.068 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:34:47.068 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:47.077 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:34:47.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd 2024-03-25T18:34:47.115 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:34:47.146 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:34:47.396 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:34:47.430 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:34:47.695 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:47.800 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:47.855 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:47.856 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T18:34:47.856 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:47.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse 2024-03-25T18:34:48.226 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:34:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:34:48.485 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:34:48.581 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:34:48.582 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:34:48.582 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:34:48.582 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:34:48.582 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:34:48.582 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:34:48.582 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:34:48.582 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:48.582 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:34:48.640 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:48.641 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:48.641 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-03-25T18:34:48.641 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:48.641 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:34:48.641 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:34:48.642 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:48.642 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:34:48.642 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:48.642 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:34:48.642 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:48.642 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 85 k 2024-03-25T18:34:48.642 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 231 k 2024-03-25T18:34:48.642 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:34:48.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:34:49.349 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 121 kB/s | 85 kB 00:00 2024-03-25T18:34:49.350 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:49.350 INFO:teuthology.orchestra.run.smithi069.stdout:Total 120 kB/s | 85 kB 00:00 2024-03-25T18:34:49.350 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:34:49.357 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:34:49.357 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:34:49.414 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:34:49.414 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:34:49.438 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:33 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:34:49.554 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:34:49.592 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:34:49.593 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-25T18:34:49.593 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:34:49.593 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-25T18:34:49.593 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:34:49.593 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:34:49.593 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-25T18:34:49.594 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-03-25T18:34:49.595 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:34:49.595 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2024-03-25T18:34:49.595 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 209 M 2024-03-25T18:34:49.595 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:34:49.656 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:50.076 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:50.117 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 249 kB/s | 32 kB 00:00 2024-03-25T18:34:50.159 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 412 kB/s | 17 kB 00:00 2024-03-25T18:34:50.201 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 829 kB/s | 34 kB 00:00 2024-03-25T18:34:50.368 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 29 MB/s | 4.7 MB 00:00 2024-03-25T18:34:50.372 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:50.372 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:50.372 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:34:50.372 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:50.372 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:50.372 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:34:50.502 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:34:50.543 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T18:34:50.584 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror 2024-03-25T18:34:50.602 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-03-25T18:34:50.736 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.0 MB/s | 803 kB 00:00 2024-03-25T18:34:50.919 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T18:34:50.986 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 788 kB/s | 195 kB 00:00 2024-03-25T18:34:51.062 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-25T18:34:51.152 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:34:51.305 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:34:51.305 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:51.305 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T18:34:51.305 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:51.305 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:34:51.305 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:34:51.305 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:51.305 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:34:51.306 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:34:51.306 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:34:51.306 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:34:51.306 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 3.1 M 2024-03-25T18:34:51.306 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 13 M 2024-03-25T18:34:51.306 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:35:00.451 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 346 kB/s | 3.1 MB 00:09 2024-03-25T18:35:00.452 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:00.452 INFO:teuthology.orchestra.run.smithi069.stdout:Total 346 kB/s | 3.1 MB 00:09 2024-03-25T18:35:00.452 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:35:00.464 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:35:00.464 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:35:00.540 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:35:00.540 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:35:00.982 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:35:01.052 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:01.077 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:01.077 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:01.077 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:35:01.077 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:01.077 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:01.078 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:02.134 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:02.135 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:02.135 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:35:02.135 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:02.135 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:02.135 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:35:02.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd 2024-03-25T18:35:02.940 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:55 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:35:03.087 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:35:03.088 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:35:03.088 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:03.088 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:35:03.088 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:35:03.088 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:35:03.088 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 171 k 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 494 k 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:35:03.729 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 268 kB/s | 171 kB 00:00 2024-03-25T18:35:03.730 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:03.730 INFO:teuthology.orchestra.run.smithi069.stdout:Total 267 kB/s | 171 kB 00:00 2024-03-25T18:35:03.730 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:35:03.738 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:35:03.739 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:35:03.796 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:35:03.796 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:35:03.943 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:35:04.086 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:04.260 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 530 kB/s | 7.4 MB 00:14 2024-03-25T18:35:04.264 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:04.264 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.5 MB/s | 37 MB 00:14 2024-03-25T18:35:04.307 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:04.333 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:04.333 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:04.498 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:04.699 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:04.699 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:04.772 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:04.772 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:04.772 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:35:04.772 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:04.772 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:04.773 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:35:05.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:35:05.407 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:05.491 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:35:05.522 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:35:05.576 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:58 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:35:05.617 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:05.659 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:35:05.730 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:35:05.731 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:35:05.731 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T18:35:05.731 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:35:05.731 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 91 M 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 498 M 2024-03-25T18:35:05.732 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:35:05.839 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:05.866 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:05.950 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:35:06.603 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:35:06.721 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:35:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:35:09.025 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:35:09.057 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:11.743 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:11.743 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:35:11.743 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:35:11.743 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:11.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:35:11.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:11.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:11.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:35:11.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:35:11.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:35:11.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:35:11.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:35:12.213 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:35:12.214 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:35:12.214 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:35:12.214 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:35:12.214 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:12.214 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:12.214 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:35:12.214 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:12.214 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:12.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:35:13.183 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:57 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:13.334 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:13.335 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:35:13.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-03-25T18:35:13.337 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:13.338 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.0 M 2024-03-25T18:35:13.338 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-03-25T18:35:13.338 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:13.816 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 206 kB/s | 24 kB 00:00 2024-03-25T18:35:13.866 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-03-25T18:35:13.917 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:35:13.951 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-25T18:35:13.975 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T18:35:14.001 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-25T18:35:14.376 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 77 kB/s | 52 kB 00:00 2024-03-25T18:35:14.409 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2024-03-25T18:35:14.435 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2024-03-25T18:35:14.660 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 262 kB/s | 59 kB 00:00 2024-03-25T18:35:14.885 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 398 kB/s | 90 kB 00:00 2024-03-25T18:35:15.069 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-25T18:35:15.094 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-google-auth-2.28.2-1.el9.noarc 282 kB/s | 203 kB 00:00 2024-03-25T18:35:15.098 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:15.099 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-25T18:35:15.099 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:15.112 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:15.112 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:15.203 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:15.205 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:15.721 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:15.890 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:35:15.936 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:35:16.000 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:35:16.046 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:35:16.087 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:35:16.180 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:35:16.289 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:35:16.349 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:35:17.069 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:35:17.142 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:35:17.180 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:35:17.228 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:35:17.250 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:18.065 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:35:18.066 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:35:18.427 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:18.428 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:18.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:35:19.263 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:19.414 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:19.414 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-03-25T18:35:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:19.416 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.8 M 2024-03-25T18:35:19.416 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 33 M 2024-03-25T18:35:19.416 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:20.056 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T18:35:20.081 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 710 kB/s | 248 kB 00:00 2024-03-25T18:35:20.198 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 494 kB/s | 58 kB 00:00 2024-03-25T18:35:20.256 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 876 kB/s | 51 kB 00:00 2024-03-25T18:35:20.315 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-03-25T18:35:20.915 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 116 kB/s | 138 kB 00:01 2024-03-25T18:35:20.916 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:20.917 INFO:teuthology.orchestra.run.smithi169.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-03-25T18:35:20.930 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:20.940 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:20.940 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:21.030 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:21.031 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:21.498 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:35:21.542 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:35:22.062 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:35:22.153 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:35:22.223 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:35:22.246 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:23.033 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:23.034 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:35:23.034 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:35:23.034 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:35:23.034 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:35:23.034 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:23.337 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:23.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-03-25T18:35:24.174 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:24.324 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:24.325 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:24.325 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:35:24.325 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:24.325 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:24.325 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:35:24.325 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:35:24.325 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:35:24.325 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:24.326 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:24.326 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:24.326 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-25T18:35:24.326 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:24.326 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 964 k 2024-03-25T18:35:24.326 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.8 M 2024-03-25T18:35:24.326 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:24.450 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 634 kB/s | 79 kB 00:00 2024-03-25T18:35:26.725 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 369 kB/s | 885 kB 00:02 2024-03-25T18:35:26.727 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:26.727 INFO:teuthology.orchestra.run.smithi169.stdout:Total 401 kB/s | 964 kB 00:02 2024-03-25T18:35:26.727 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:26.751 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:26.751 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:26.827 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:26.827 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:27.021 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:27.177 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:35:27.250 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:27.948 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:27.948 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:35:28.269 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:35:28.269 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:28.269 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:28.269 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:35:28.269 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:28.269 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:28.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-volume 2024-03-25T18:35:29.174 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:29.327 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:29.327 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:29.327 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T18:35:29.327 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:29.327 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:29.327 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:35:29.327 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:35:29.328 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:35:29.328 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:29.328 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:29.328 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:29.328 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-25T18:35:29.328 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:29.328 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 332 k 2024-03-25T18:35:29.328 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.5 M 2024-03-25T18:35:29.329 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:29.455 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 544 kB/s | 69 kB 00:00 2024-03-25T18:35:29.664 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 787 kB/s | 264 kB 00:00 2024-03-25T18:35:29.665 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:29.665 INFO:teuthology.orchestra.run.smithi169.stdout:Total 985 kB/s | 332 kB 00:00 2024-03-25T18:35:29.665 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:29.674 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:29.674 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:35:30.100 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:30.115 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:30.115 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:30.115 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:35:30.115 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:30.903 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:35:31.249 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:35:31.249 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:31.249 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:31.249 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:31.249 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:35:31.250 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:31.250 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:31.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-03-25T18:35:32.185 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:32.335 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:35:32.336 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:32.337 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 127 k 2024-03-25T18:35:32.337 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 456 k 2024-03-25T18:35:32.337 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:32.845 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 249 kB/s | 127 kB 00:00 2024-03-25T18:35:32.846 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:32.846 INFO:teuthology.orchestra.run.smithi169.stdout:Total 248 kB/s | 127 kB 00:00 2024-03-25T18:35:32.846 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:32.852 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:32.852 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:32.909 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:32.909 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:33.050 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:33.169 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:33.672 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:33.957 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:33.957 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:33.957 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:33.958 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:33.958 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:33.958 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:34.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-03-25T18:35:34.766 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:34.866 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:34.919 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:34.920 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:35:34.921 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:34.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-03-25T18:35:35.550 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:35.700 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:35.700 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:35:35.701 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:35.702 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 31 k 2024-03-25T18:35:35.702 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 140 k 2024-03-25T18:35:35.702 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:36.224 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 60 kB/s | 31 kB 00:00 2024-03-25T18:35:36.224 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:36.224 INFO:teuthology.orchestra.run.smithi169.stdout:Total 60 kB/s | 31 kB 00:00 2024-03-25T18:35:36.224 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:36.228 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:36.228 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:36.258 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:36.258 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:36.375 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:36.481 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:36.865 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:37.130 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:37.130 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:37.130 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:37.130 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:37.130 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:37.130 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:37.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-03-25T18:35:37.853 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:37.949 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:38.001 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:38.002 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:35:38.002 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:38.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-03-25T18:35:38.622 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:22 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:38.719 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:38.772 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:38.773 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:35:38.773 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:38.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-03-25T18:35:39.385 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:39.483 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:39.535 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:39.536 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:35:39.536 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:39.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-03-25T18:35:40.188 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:40.294 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:40.345 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:40.346 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:35:40.346 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:40.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-03-25T18:35:40.982 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:41.078 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:41.130 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:41.131 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:35:41.132 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:41.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-03-25T18:35:41.741 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:41.838 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:41.890 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:41.891 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T18:35:41.891 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:41.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-03-25T18:35:42.503 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:42.652 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:42.652 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:35:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:42.654 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 85 k 2024-03-25T18:35:42.654 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 231 k 2024-03-25T18:35:42.654 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:43.552 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 95 kB/s | 85 kB 00:00 2024-03-25T18:35:43.553 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:43.553 INFO:teuthology.orchestra.run.smithi169.stdout:Total 95 kB/s | 85 kB 00:00 2024-03-25T18:35:43.553 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:43.562 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:43.562 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:43.620 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:43.620 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:43.759 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:43.859 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:44.311 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:44.590 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:44.590 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:44.590 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:44.590 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:44.590 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:44.590 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:44.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-03-25T18:35:45.408 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:45.559 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:45.560 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:45.560 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:35:45.560 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:45.560 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:45.560 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:35:45.560 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:45.561 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:45.561 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:45.561 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:35:45.561 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:45.561 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.1 M 2024-03-25T18:35:45.561 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 13 M 2024-03-25T18:35:45.561 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:46.685 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 561 kB/s | 22 MB 00:40 2024-03-25T18:35:52.255 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 472 kB/s | 3.1 MB 00:06 2024-03-25T18:35:52.256 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:52.256 INFO:teuthology.orchestra.run.smithi169.stdout:Total 472 kB/s | 3.1 MB 00:06 2024-03-25T18:35:52.256 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:52.268 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:52.268 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:52.344 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:52.345 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:52.847 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:52.848 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:52.848 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:35:52.848 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:52.848 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:52.848 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:54.063 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:54.063 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:54.063 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:54.063 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:54.063 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:54.063 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:54.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-03-25T18:35:54.893 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:55.044 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:35:55.045 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:55.046 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 171 k 2024-03-25T18:35:55.046 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 494 k 2024-03-25T18:35:55.046 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:35:56.034 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 174 kB/s | 171 kB 00:00 2024-03-25T18:35:56.034 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:56.035 INFO:teuthology.orchestra.run.smithi169.stdout:Total 173 kB/s | 171 kB 00:00 2024-03-25T18:35:56.035 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:35:56.043 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:35:56.043 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:35:56.100 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:35:56.101 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:35:56.256 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:35:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:56.830 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:57.096 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:57.096 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:57.096 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:35:57.096 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:57.096 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:57.096 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:35:57.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:35:57.900 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:35:58.052 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:35:58.053 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-25T18:35:58.054 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:35:58.054 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 91 M 2024-03-25T18:35:58.054 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 498 M 2024-03-25T18:35:58.054 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:37:03.728 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 350 kB/s | 22 MB 01:05 2024-03-25T18:37:57.418 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 406 kB/s | 68 MB 02:51 2024-03-25T18:37:57.419 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:37:57.419 INFO:teuthology.orchestra.run.smithi069.stdout:Total 540 kB/s | 91 MB 02:51 2024-03-25T18:37:57.419 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:37:57.423 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:37:57.423 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:37:58.012 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:37:58.012 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:38:01.122 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:38:08.233 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:38:08.309 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:08.761 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 534 kB/s | 68 MB 02:10 2024-03-25T18:38:08.762 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:08.762 INFO:teuthology.orchestra.run.smithi169.stdout:Total 710 kB/s | 91 MB 02:10 2024-03-25T18:38:08.763 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:38:08.767 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:38:08.767 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:38:09.358 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:38:09.359 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:38:11.673 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:11.674 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:38:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:38:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:38:11.942 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:38:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:38:11.942 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:38:12.149 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:38:12.468 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:38:12.717 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:05 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:38:12.873 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:38:12.874 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:38:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T18:38:12.874 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 416 M 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.7 G 2024-03-25T18:38:12.875 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:38:19.626 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:38:19.787 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:23.144 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:23.144 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:38:23.405 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:38:23.406 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:38:23.406 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:38:23.406 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:23.406 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:23.406 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:38:23.406 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:38:23.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:38:24.207 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:08:08 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:38:24.360 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:38:24.361 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:38:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:38:24.361 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 416 M 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.7 G 2024-03-25T18:38:24.362 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:51:04.627 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 560 kB/s | 416 MB 12:40 2024-03-25T18:51:04.627 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:04.627 INFO:teuthology.orchestra.run.smithi169.stdout:Total 560 kB/s | 416 MB 12:40 2024-03-25T18:51:04.628 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:51:04.630 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:51:04.630 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:51:07.109 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:51:07.110 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:51:49.443 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:51:49.554 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:51:54.648 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:51:54.907 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:51:54.907 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:51:54.907 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:51:54.907 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:54.907 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:51:54.907 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:51:55.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:51:55.744 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:21:39 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:51:55.896 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:51:55.897 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:51:55.897 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:51:55.897 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:51:55.897 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:51:55.897 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:51:55.897 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:51:55.897 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:51:55.897 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T18:51:55.898 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:51:55.898 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:51:55.898 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.4 M 2024-03-25T18:51:55.898 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 6.4 M 2024-03-25T18:51:55.898 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:52:02.365 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 226 kB/s | 1.4 MB 00:06 2024-03-25T18:52:02.365 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:02.366 INFO:teuthology.orchestra.run.smithi169.stdout:Total 226 kB/s | 1.4 MB 00:06 2024-03-25T18:52:02.366 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:52:02.370 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:52:02.370 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:52:02.417 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:52:02.417 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:52:02.693 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:52:02.780 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:03.163 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:03.436 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:03.436 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:52:03.436 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:52:03.436 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:03.436 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:52:03.436 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:52:03.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:52:04.184 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:21:48 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:52:04.333 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:52:04.334 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:52:04.334 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 176 M 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 753 M 2024-03-25T18:52:04.335 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:58:37.264 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 348 kB/s | 416 MB 20:24 2024-03-25T18:58:37.264 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:58:37.264 INFO:teuthology.orchestra.run.smithi069.stdout:Total 348 kB/s | 416 MB 20:24 2024-03-25T18:58:37.265 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:58:37.267 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:58:37.268 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:58:39.727 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:58:39.728 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:58:45.403 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 448 kB/s | 176 MB 06:41 2024-03-25T18:58:45.403 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:58:45.404 INFO:teuthology.orchestra.run.smithi169.stdout:Total 448 kB/s | 176 MB 06:41 2024-03-25T18:58:45.404 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T18:58:45.407 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T18:58:45.407 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T18:58:46.470 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T18:58:46.470 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T18:59:04.463 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T18:59:04.567 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:59:09.204 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:59:09.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:59:09.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:59:09.460 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T18:59:09.460 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:09.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:59:09.460 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T18:59:09.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:59:10.253 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:28:54 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T18:59:10.403 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T18:59:10.404 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T18:59:10.405 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T18:59:10.406 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 672 M 2024-03-25T18:59:10.406 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 3.1 G 2024-03-25T18:59:10.406 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T18:59:21.811 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:59:21.914 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:59:30.780 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:59:31.033 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:59:31.033 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:59:31.033 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:59:31.034 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:31.034 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:59:31.034 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:59:31.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:59:31.846 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:29:24 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:59:31.994 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:59:31.995 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:59:31.995 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T18:59:31.995 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:59:31.995 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:59:31.995 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:59:31.995 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:59:31.995 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:59:31.995 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-03-25T18:59:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:59:31.996 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:59:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.4 M 2024-03-25T18:59:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 6.4 M 2024-03-25T18:59:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T18:59:35.618 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 403 kB/s | 1.4 MB 00:03 2024-03-25T18:59:35.619 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:35.619 INFO:teuthology.orchestra.run.smithi069.stdout:Total 403 kB/s | 1.4 MB 00:03 2024-03-25T18:59:35.619 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T18:59:35.624 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T18:59:35.624 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T18:59:35.669 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T18:59:35.669 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T18:59:35.942 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T18:59:36.049 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:59:36.419 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:59:36.688 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:59:36.688 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:59:36.688 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T18:59:36.688 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:36.688 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:59:36.688 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T18:59:36.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:59:37.431 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:29:30 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T18:59:37.578 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T18:59:37.579 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:59:37.579 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T18:59:37.579 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:59:37.579 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 176 M 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 753 M 2024-03-25T18:59:37.580 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:05:27.375 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 514 kB/s | 176 MB 05:49 2024-03-25T19:05:27.376 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:05:27.376 INFO:teuthology.orchestra.run.smithi069.stdout:Total 514 kB/s | 176 MB 05:49 2024-03-25T19:05:27.376 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:05:27.379 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:05:27.379 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:05:28.444 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:05:28.444 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:05:46.755 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:05:46.920 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:51.440 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:51.698 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:51.698 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:05:51.698 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:05:51.698 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:51.698 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:05:51.699 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:05:51.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:05:52.471 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:35:45 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:05:52.619 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:05:52.620 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:05:52.620 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:05:52.620 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:05:52.620 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:05:52.620 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:05:52.621 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:05:52.621 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:05:52.621 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:05:52.621 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:05:52.621 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:05:52.621 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 672 M 2024-03-25T19:05:52.621 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 3.1 G 2024-03-25T19:05:52.621 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:28:09.132 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 396 kB/s | 672 MB 28:58 2024-03-25T19:28:09.133 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:09.133 INFO:teuthology.orchestra.run.smithi169.stdout:Total 396 kB/s | 672 MB 28:58 2024-03-25T19:28:09.133 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:28:09.137 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:28:09.137 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:28:13.111 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:28:13.111 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:28:25.865 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 509 kB/s | 672 MB 22:33 2024-03-25T19:28:25.866 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:25.866 INFO:teuthology.orchestra.run.smithi069.stdout:Total 509 kB/s | 672 MB 22:33 2024-03-25T19:28:25.866 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:28:25.870 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:28:25.871 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:28:29.841 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:28:29.841 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:29:24.166 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:29:24.330 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:34.437 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:34.685 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:34.686 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:29:34.686 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:29:34.686 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:34.686 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:29:34.686 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:29:34.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:29:35.559 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:59:19 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:29:35.659 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:29:35.717 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:29:35.718 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T19:29:35.718 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:29:35.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:29:36.378 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:59:20 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:29:36.533 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:29:36.534 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:29:36.534 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:29:36.534 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 20 M 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 103 M 2024-03-25T19:29:36.535 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:29:40.388 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:29:41.008 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:45.658 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 2.2 MB/s | 20 MB 00:09 2024-03-25T19:29:45.659 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:45.659 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.2 MB/s | 20 MB 00:09 2024-03-25T19:29:45.660 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:29:45.664 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:29:45.664 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:29:45.814 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:29:45.814 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:29:48.052 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:29:48.165 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:49.193 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:49.453 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:49.453 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:29:49.453 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:29:49.453 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:49.453 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:29:49.453 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:29:49.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:29:49.785 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:50.071 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:50.071 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:29:50.071 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:29:50.071 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:50.071 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:29:50.071 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:29:50.231 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:59:34 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:29:50.353 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:29:50.381 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:29:50.381 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:29:50.381 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:29:50.381 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:29:50.381 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:29:50.381 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:29:50.381 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:29:50.382 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:29:50.382 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:29:50.382 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:29:50.382 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:29:50.382 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 48 M 2024-03-25T19:29:50.382 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 173 M 2024-03-25T19:29:50.382 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:29:50.922 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:59:43 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:29:51.018 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:29:51.070 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:29:51.071 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-03-25T19:29:51.071 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:29:51.128 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:29:51.688 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:59:44 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:29:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 20 M 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 103 M 2024-03-25T19:29:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:30:19.315 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 734 kB/s | 20 MB 00:27 2024-03-25T19:30:19.316 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:19.317 INFO:teuthology.orchestra.run.smithi069.stdout:Total 734 kB/s | 20 MB 00:27 2024-03-25T19:30:19.317 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:30:19.320 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:30:19.321 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:30:19.471 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:30:19.472 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:30:21.690 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:30:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:22.874 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:23.143 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:23.144 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:30:23.144 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:30:23.144 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:30:23.144 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:30:23.144 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:30:23.324 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:30:23.893 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:00:16 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:30:24.042 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:30:24.046 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:30:24.046 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:30:24.046 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:30:24.046 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:30:24.046 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:30:24.046 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:30:24.046 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:30:24.046 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:30:24.047 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:30:24.047 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:30:24.047 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 48 M 2024-03-25T19:30:24.047 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 173 M 2024-03-25T19:30:24.047 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:30:26.848 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 48 MB 00:36 2024-03-25T19:30:26.849 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:26.849 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.3 MB/s | 48 MB 00:36 2024-03-25T19:30:26.849 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:30:26.853 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:30:26.853 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:30:27.166 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:30:27.166 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:30:31.941 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:30:32.056 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:33.484 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:33.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:33.750 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:30:33.751 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:30:33.751 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:30:33.751 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:30:33.751 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:30:33.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:30:34.534 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:00:18 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:30:34.683 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:30:34.683 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:30:34.683 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:30:34.683 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:30:34.683 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:30:34.684 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:30:34.684 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:30:34.684 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:30:34.684 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:30:34.684 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:30:34.684 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:30:34.684 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 99 M 2024-03-25T19:30:34.684 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 449 M 2024-03-25T19:30:34.685 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:30:59.513 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 48 MB 00:35 2024-03-25T19:30:59.514 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:59.514 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.3 MB/s | 48 MB 00:35 2024-03-25T19:30:59.514 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:30:59.518 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:30:59.518 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:30:59.830 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:30:59.831 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:31:04.573 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:31:04.704 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:06.006 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:06.278 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:06.278 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:31:06.278 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:31:06.278 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:06.279 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:31:06.279 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:31:06.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:31:07.010 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:01:00 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:31:07.157 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:31:07.158 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:31:07.158 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:31:07.158 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:31:07.158 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:31:07.158 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:31:07.158 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:31:07.159 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:31:07.159 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:31:07.159 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:31:07.159 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:31:07.159 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 99 M 2024-03-25T19:31:07.159 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 449 M 2024-03-25T19:31:07.159 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:31:59.315 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 99 MB 01:24 2024-03-25T19:31:59.315 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:59.316 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.2 MB/s | 99 MB 01:24 2024-03-25T19:31:59.316 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:31:59.320 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:31:59.320 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:31:59.933 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:31:59.934 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:32:10.236 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:32:10.337 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:13.330 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:13.587 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:13.587 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:32:13.587 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:32:13.587 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:13.587 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:32:13.587 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:32:13.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:32:14.381 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:01:58 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:32:14.530 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 287 M 2024-03-25T19:32:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.2 G 2024-03-25T19:32:14.532 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:32:43.270 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 99 MB 01:36 2024-03-25T19:32:43.270 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:43.271 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.0 MB/s | 99 MB 01:36 2024-03-25T19:32:43.271 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:32:43.274 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:32:43.274 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:32:43.883 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:32:43.884 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:32:54.137 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:32:54.242 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:57.017 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:57.284 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:57.284 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:32:57.284 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:32:57.284 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:57.284 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:32:57.284 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:32:57.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:32:58.032 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:02:51 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:32:58.181 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:32:58.182 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:32:58.182 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:32:58.182 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 287 M 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.2 G 2024-03-25T19:32:58.183 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:36:44.671 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 287 MB 04:30 2024-03-25T19:36:44.672 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:44.672 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.1 MB/s | 287 MB 04:30 2024-03-25T19:36:44.672 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:36:44.676 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:36:44.676 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:36:46.389 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:36:46.389 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:36:48.972 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 287 MB 03:50 2024-03-25T19:36:48.973 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:48.974 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.2 MB/s | 287 MB 03:50 2024-03-25T19:36:48.974 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:36:48.977 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:36:48.977 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:36:50.686 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:36:50.687 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:37:16.073 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:37:16.627 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:20.148 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:37:20.663 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:22.936 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:22.936 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:22.936 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:37:22.936 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:22.936 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:22.937 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:37:23.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:37:23.742 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:07:07 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:37:23.892 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:37:23.893 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:23.894 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 5.9 M 2024-03-25T19:37:23.894 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 28 M 2024-03-25T19:37:23.894 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:37:24.635 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:24.922 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:24.922 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:24.922 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:37:24.922 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:24.922 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:24.922 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:37:25.131 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:37:25.698 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:07:18 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:37:25.846 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:37:25.846 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:37:25.847 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:25.848 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 5.9 M 2024-03-25T19:37:25.848 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 28 M 2024-03-25T19:37:25.848 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:37:30.002 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 996 kB/s | 5.9 MB 00:06 2024-03-25T19:37:30.003 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:30.004 INFO:teuthology.orchestra.run.smithi169.stdout:Total 996 kB/s | 5.9 MB 00:06 2024-03-25T19:37:30.004 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:37:30.008 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:37:30.008 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:37:30.080 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:37:30.081 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:37:30.816 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:37:30.926 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:31.500 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:31.764 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:31.764 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:31.764 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:37:31.764 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:31.765 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:31.765 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:37:31.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:37:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:07:16 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:37:32.704 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:37:32.704 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:37:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:32.706 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 109 k 2024-03-25T19:37:32.706 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 440 k 2024-03-25T19:37:32.706 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:37:33.456 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 145 kB/s | 109 kB 00:00 2024-03-25T19:37:33.456 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:33.456 INFO:teuthology.orchestra.run.smithi169.stdout:Total 145 kB/s | 109 kB 00:00 2024-03-25T19:37:33.457 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:37:33.460 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:37:33.460 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:37:33.493 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:37:33.494 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:37:33.608 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:37:33.712 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:34.112 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:34.386 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:34.386 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:34.386 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:37:34.387 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:34.387 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:34.387 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:37:34.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:37:35.182 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:07:19 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:37:35.330 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:37:35.330 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 5.4 M 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 25 M 2024-03-25T19:37:35.331 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:37:38.688 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 474 kB/s | 5.9 MB 00:12 2024-03-25T19:37:38.689 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:38.689 INFO:teuthology.orchestra.run.smithi069.stdout:Total 474 kB/s | 5.9 MB 00:12 2024-03-25T19:37:38.690 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:37:38.694 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:37:38.694 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:37:38.767 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:37:38.767 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:37:39.508 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:37:39.619 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:40.177 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:40.448 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:40.448 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:40.448 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:37:40.448 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:40.448 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:40.448 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:37:40.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:37:41.189 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:07:34 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:37:41.341 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:37:41.342 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:37:41.342 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:41.342 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:37:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:37:41.342 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:41.342 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:37:41.343 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:41.343 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:37:41.343 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:41.343 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 109 k 2024-03-25T19:37:41.343 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 440 k 2024-03-25T19:37:41.343 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:37:42.232 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 122 kB/s | 109 kB 00:00 2024-03-25T19:37:42.233 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:42.233 INFO:teuthology.orchestra.run.smithi069.stdout:Total 122 kB/s | 109 kB 00:00 2024-03-25T19:37:42.233 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:37:42.237 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:37:42.237 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:37:42.271 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:37:42.271 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:37:42.395 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:37:42.497 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:42.874 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:42.999 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 715 kB/s | 5.4 MB 00:07 2024-03-25T19:37:43.000 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:43.001 INFO:teuthology.orchestra.run.smithi169.stdout:Total 715 kB/s | 5.4 MB 00:07 2024-03-25T19:37:43.001 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:37:43.005 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:37:43.005 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:37:43.074 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:37:43.075 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:37:43.138 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:43.139 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:37:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:43.139 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:43.139 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:37:43.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:37:43.744 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:37:43.841 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:43.882 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:07:36 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:37:44.031 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:37:44.031 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:44.031 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:37:44.031 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:44.031 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:37:44.031 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:37:44.031 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:44.032 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:37:44.032 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:44.032 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:37:44.032 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:44.032 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 5.4 M 2024-03-25T19:37:44.032 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 25 M 2024-03-25T19:37:44.032 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:37:44.391 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:44.652 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:44.652 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:37:44.652 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:44.652 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:37:44.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2-debuginfo 2024-03-25T19:37:45.453 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:07:29 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:37:45.600 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:37:45.601 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:37:45.602 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 44 M 2024-03-25T19:37:45.602 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 253 M 2024-03-25T19:37:45.602 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:37:51.568 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 727 kB/s | 5.4 MB 00:07 2024-03-25T19:37:51.569 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:51.569 INFO:teuthology.orchestra.run.smithi069.stdout:Total 727 kB/s | 5.4 MB 00:07 2024-03-25T19:37:51.570 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:37:51.574 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:37:51.574 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:37:51.641 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:37:51.642 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:37:52.325 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:37:52.433 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:52.980 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:53.251 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:53.251 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:53.251 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:37:53.251 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:53.251 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:53.251 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:37:53.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2-debuginfo 2024-03-25T19:37:54.012 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:07:47 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:37:54.160 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:37:54.161 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:54.161 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:37:54.161 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:54.161 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:37:54.161 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:37:54.161 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:54.161 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:37:54.161 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:37:54.162 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:37:54.162 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:37:54.162 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 44 M 2024-03-25T19:37:54.162 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 253 M 2024-03-25T19:37:54.162 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:38:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.8 MB/s | 44 MB 00:24 2024-03-25T19:38:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.8 MB/s | 44 MB 00:24 2024-03-25T19:38:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:38:19.049 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:38:19.049 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:38:19.348 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:38:19.348 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:38:24.420 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:38:24.509 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:26.296 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:26.585 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:26.585 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:38:26.585 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:38:26.585 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:26.585 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:38:26.585 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:38:26.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:38:27.342 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:08:20 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:38:27.488 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:38:27.489 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:38:27.490 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 52 M 2024-03-25T19:38:27.490 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 308 M 2024-03-25T19:38:27.490 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:38:29.121 INFO:teuthology.orchestra.run.smithi169.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.0 MB/s | 44 MB 00:43 2024-03-25T19:38:29.122 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:29.123 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.0 MB/s | 44 MB 00:43 2024-03-25T19:38:29.123 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:38:29.126 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:38:29.127 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:38:29.426 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:38:29.426 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:38:34.501 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:38:34.604 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:36.401 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:36.668 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:36.668 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:38:36.668 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:38:36.668 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:36.668 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:38:36.668 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:38:36.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:38:37.472 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:08:21 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:38:37.622 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:38:37.623 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:38:37.624 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:38:37.624 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 52 M 2024-03-25T19:38:37.624 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 308 M 2024-03-25T19:38:37.624 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:38:52.368 INFO:teuthology.orchestra.run.smithi069.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.1 MB/s | 52 MB 00:24 2024-03-25T19:38:52.368 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:52.369 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.1 MB/s | 52 MB 00:24 2024-03-25T19:38:52.369 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:38:52.373 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:38:52.373 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:38:52.719 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:38:52.719 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:38:58.704 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:38:58.899 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:01.039 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:01.303 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:01.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:01.304 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:39:01.304 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:01.304 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:01.304 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:39:01.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:39:02.069 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:08:55 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:39:02.220 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:39:02.220 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:02.222 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 615 k 2024-03-25T19:39:02.222 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.1 M 2024-03-25T19:39:02.222 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:39:03.162 INFO:teuthology.orchestra.run.smithi069.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 654 kB/s | 615 kB 00:00 2024-03-25T19:39:03.162 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:03.162 INFO:teuthology.orchestra.run.smithi069.stdout:Total 653 kB/s | 615 kB 00:00 2024-03-25T19:39:03.163 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:39:03.166 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:39:03.166 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:39:03.207 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:39:03.208 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:39:03.376 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:39:03.494 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:03.646 INFO:teuthology.orchestra.run.smithi169.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.0 MB/s | 52 MB 00:26 2024-03-25T19:39:03.647 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:03.647 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.0 MB/s | 52 MB 00:26 2024-03-25T19:39:03.647 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:39:03.651 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:39:03.651 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:39:03.899 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:03.996 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:39:03.997 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:39:04.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:04.168 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:04.168 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:39:04.168 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:04.168 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:04.168 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:39:04.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:39:04.936 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:08:57 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:39:05.088 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:39:05.089 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:05.089 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 771 k 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.7 M 2024-03-25T19:39:05.090 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:39:05.956 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 890 kB/s | 771 kB 00:00 2024-03-25T19:39:05.957 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:05.957 INFO:teuthology.orchestra.run.smithi069.stdout:Total 889 kB/s | 771 kB 00:00 2024-03-25T19:39:05.957 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:39:05.961 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:39:05.961 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:39:05.999 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:39:06.000 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:39:06.184 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:39:06.285 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:06.672 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:06.960 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:06.960 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:06.960 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:39:06.960 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:06.960 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:06.960 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:39:07.163 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:39:07.718 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:09:00 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:39:07.866 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:39:07.867 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:07.867 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:39:07.867 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:07.867 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:39:07.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:39:07.867 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:07.867 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:39:07.867 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:07.868 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:39:07.868 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:07.868 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.1 M 2024-03-25T19:39:07.868 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 4.3 M 2024-03-25T19:39:07.868 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:39:09.159 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 891 kB/s | 1.1 MB 00:01 2024-03-25T19:39:09.160 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:09.160 INFO:teuthology.orchestra.run.smithi069.stdout:Total 890 kB/s | 1.1 MB 00:01 2024-03-25T19:39:09.161 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:39:09.164 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:39:09.165 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:39:09.208 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:39:09.208 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:39:09.453 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:39:09.541 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:09.924 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:09.983 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:39:10.120 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:10.191 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:39:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:10.191 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:39:10.389 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:39:10.946 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:09:03 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:39:11.093 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:39:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:11.095 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 337 k 2024-03-25T19:39:11.095 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.0 M 2024-03-25T19:39:11.095 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:39:12.029 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 361 kB/s | 337 kB 00:00 2024-03-25T19:39:12.030 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:12.030 INFO:teuthology.orchestra.run.smithi069.stdout:Total 361 kB/s | 337 kB 00:00 2024-03-25T19:39:12.030 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:39:12.034 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:39:12.034 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:39:12.070 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:39:12.070 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:39:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:39:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:12.580 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:12.580 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:12.581 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:39:12.581 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:12.581 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:12.581 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:39:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:12.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:39:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:12.976 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:39:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:12.976 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:39:13.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:39:13.373 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:08:57 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:39:13.521 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:39:13.522 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:13.522 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 615 k 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.1 M 2024-03-25T19:39:13.523 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:39:13.729 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:09:06 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:39:13.877 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:39:13.878 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:13.878 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:39:13.878 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:13.878 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:39:13.878 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:39:13.879 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:13.879 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:39:13.879 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:13.879 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:39:13.879 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:13.879 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 509 k 2024-03-25T19:39:13.879 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.5 M 2024-03-25T19:39:13.879 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:39:14.240 INFO:teuthology.orchestra.run.smithi169.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 858 kB/s | 615 kB 00:00 2024-03-25T19:39:14.240 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:14.241 INFO:teuthology.orchestra.run.smithi169.stdout:Total 856 kB/s | 615 kB 00:00 2024-03-25T19:39:14.241 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:39:14.245 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:39:14.245 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:39:14.284 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:39:14.284 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:39:14.454 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:39:14.573 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:14.633 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 675 kB/s | 509 kB 00:00 2024-03-25T19:39:14.633 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:14.634 INFO:teuthology.orchestra.run.smithi069.stdout:Total 674 kB/s | 509 kB 00:00 2024-03-25T19:39:14.634 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:39:14.637 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:39:14.637 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:39:14.675 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:39:14.675 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:39:14.834 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:39:14.940 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:14.980 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:15.290 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:39:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:15.290 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:39:15.337 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:15.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:39:15.618 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:15.618 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:15.618 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:39:15.618 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:15.618 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:15.618 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:39:15.809 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:39:16.066 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:09:00 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:39:16.219 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:39:16.220 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:16.220 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:39:16.220 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:16.220 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:39:16.220 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:39:16.220 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:16.220 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:39:16.220 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:16.221 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:39:16.221 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:16.221 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 771 k 2024-03-25T19:39:16.221 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.7 M 2024-03-25T19:39:16.221 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:39:16.370 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:09:09 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:39:16.519 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:39:16.520 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:39:16.520 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:16.520 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:39:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:16.521 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:39:16.521 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:16.521 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:39:16.521 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:16.521 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 52 M 2024-03-25T19:39:16.521 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 298 M 2024-03-25T19:39:16.521 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:39:17.396 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 656 kB/s | 771 kB 00:01 2024-03-25T19:39:17.396 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:17.396 INFO:teuthology.orchestra.run.smithi169.stdout:Total 655 kB/s | 771 kB 00:01 2024-03-25T19:39:17.397 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:39:17.401 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:39:17.401 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:39:17.443 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:39:17.444 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:39:17.654 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:39:17.777 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:18.164 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:18.429 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:18.429 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:18.429 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:39:18.429 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:18.429 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:18.429 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:39:18.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:39:19.212 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:09:03 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:39:19.361 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:19.362 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.1 M 2024-03-25T19:39:19.363 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 4.3 M 2024-03-25T19:39:19.363 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:39:20.566 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 957 kB/s | 1.1 MB 00:01 2024-03-25T19:39:20.566 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:20.567 INFO:teuthology.orchestra.run.smithi169.stdout:Total 955 kB/s | 1.1 MB 00:01 2024-03-25T19:39:20.567 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:39:20.571 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:39:20.571 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:39:20.615 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:39:20.616 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:39:20.858 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:39:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:21.355 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:21.626 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:21.626 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:21.626 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:39:21.626 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:21.626 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:21.626 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:39:21.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:39:22.410 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:09:06 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:39:22.561 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:39:22.562 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:22.562 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:39:22.562 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:22.562 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:39:22.562 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:39:22.562 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:22.563 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:39:22.563 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:22.563 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:39:22.563 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:22.563 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 337 k 2024-03-25T19:39:22.563 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.0 M 2024-03-25T19:39:22.563 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:39:23.078 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 656 kB/s | 337 kB 00:00 2024-03-25T19:39:23.078 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:23.079 INFO:teuthology.orchestra.run.smithi169.stdout:Total 654 kB/s | 337 kB 00:00 2024-03-25T19:39:23.079 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:39:23.082 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:39:23.082 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:39:23.120 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:39:23.121 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:39:23.252 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 7.7 MB/s | 52 MB 00:06 2024-03-25T19:39:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Total 7.7 MB/s | 52 MB 00:06 2024-03-25T19:39:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:39:23.257 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:39:23.257 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:39:23.258 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:39:23.377 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:23.592 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:39:23.592 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:24.052 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:24.052 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:24.052 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:39:24.052 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:24.052 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:24.052 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:39:24.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:39:24.843 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:09:08 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:39:24.994 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:39:24.994 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 509 k 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.5 M 2024-03-25T19:39:24.995 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:39:25.289 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.7 MB/s | 509 kB 00:00 2024-03-25T19:39:25.290 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:25.290 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.7 MB/s | 509 kB 00:00 2024-03-25T19:39:25.290 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:39:25.293 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:39:25.293 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:39:25.330 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:39:25.331 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:39:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:39:25.596 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:25.970 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:26.261 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:26.261 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:26.261 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:39:26.261 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:26.261 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:26.261 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:39:26.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:39:27.042 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:09:11 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:39:27.196 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:39:27.197 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:27.197 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:39:27.197 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:27.197 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:39:27.197 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:27.197 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:27.197 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:39:27.197 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:27.198 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:39:27.198 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:27.198 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 52 M 2024-03-25T19:39:27.198 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 298 M 2024-03-25T19:39:27.198 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:39:29.558 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:39:29.824 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:31.850 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:32.121 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:32.121 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:32.121 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:39:32.121 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:32.121 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:32.121 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:39:32.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:39:32.878 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 1:09:25 ago on Mon 25 Mar 2024 06:30:07 PM UTC. 2024-03-25T19:39:33.027 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-25T19:39:33.027 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:33.028 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.3 M 2024-03-25T19:39:33.029 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 5.5 M 2024-03-25T19:39:33.029 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-25T19:39:33.345 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 4.0 MB/s | 1.3 MB 00:00 2024-03-25T19:39:33.346 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:33.346 INFO:teuthology.orchestra.run.smithi069.stdout:Total 3.9 MB/s | 1.3 MB 00:00 2024-03-25T19:39:33.346 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-25T19:39:33.349 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-25T19:39:33.350 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-25T19:39:33.375 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 8.3 MB/s | 52 MB 00:06 2024-03-25T19:39:33.377 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:33.377 INFO:teuthology.orchestra.run.smithi169.stdout:Total 8.3 MB/s | 52 MB 00:06 2024-03-25T19:39:33.377 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:39:33.381 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:39:33.381 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:39:33.391 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-25T19:39:33.391 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-25T19:39:33.626 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-25T19:39:33.716 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:39:33.716 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:39:33.719 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:34.119 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:34.379 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:34.380 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:34.380 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-25T19:39:34.380 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:34.380 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:34.380 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-25T19:39:34.570 DEBUG:teuthology.parallel:result is None 2024-03-25T19:39:39.618 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:39:39.724 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:41.930 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:42.199 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:42.200 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:42.200 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:39:42.200 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:42.200 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:42.200 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:39:42.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:39:42.990 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:09:26 ago on Mon 25 Mar 2024 06:30:16 PM UTC. 2024-03-25T19:39:43.148 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.3 M 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 5.5 M 2024-03-25T19:39:43.149 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T19:39:44.219 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 1.3 MB 00:01 2024-03-25T19:39:44.220 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:44.220 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.2 MB/s | 1.3 MB 00:01 2024-03-25T19:39:44.220 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T19:39:44.224 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T19:39:44.224 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T19:39:44.269 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T19:39:44.269 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T19:39:44.533 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T19:39:44.643 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:45.084 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:45.358 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T19:39:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:45.358 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T19:39:45.595 DEBUG:teuthology.parallel:result is None 2024-03-25T19:39:45.596 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:39:45.816 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:39:45.840 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:45.840 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:45.840 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:39:45.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:39:45.983 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:39:46.003 INFO:teuthology.orchestra.run.smithi169.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:46.004 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:46.004 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:39:46.005 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:39:46.006 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:46.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:39:46.037 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:39:46.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:39:46.071 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:39:46.071 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:46.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:39:46.107 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:39:46.177 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:39:46.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:39:46.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:39:46.271 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:39:46.271 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:46.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:39:46.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:39:46.369 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:39:46.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:39:46.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:39:46.463 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:39:46.464 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:46.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:39:46.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:39:46.561 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:39:46.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:39:46.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:39:46.655 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:39:46.739 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:39:46.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2024-03-25T19:39:46.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /var/log/ceph 2024-03-25T19:39:46.768 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:39:46.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:39:46.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:39:46.838 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:39:46.838 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:39:46.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:39:46.917 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:39:46.917 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:39:46.917 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621324', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIb/u/8YPBEaSOyeWaG1lD14x3+nk7rRSq7+W3fk8olAzDvJQ2CW4ABBgm3sDGoDCeheiNDR6yrJ/Zpl13Ne74E=', 'smithi169.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1vVUhWlSt6/30FLLdt0/GWS0KE8ZSYrh3OW3eK3B4lA2hSUSqcIhGCfPLa8vPgV7QJ24N99OVrmjZrrp019GA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:39:46.918 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:39:46.949 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:39:46.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:39:47.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:39:47.030 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:47.030 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:39:47.084 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:39:47.084 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-25 18:35:01.350486229 +0000 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-25 18:29:12.070572958 +0000 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-25 18:29:12.070572958 +0000 2024-03-25T19:39:47.139 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-25 18:29:12.070572958 +0000 2024-03-25T19:39:47.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:39:47.204 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-25T19:39:47.204 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-25T19:39:47.204 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000207651 s, 2.5 MB/s 2024-03-25T19:39:47.205 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:39:47.261 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-03-25T19:39:47.315 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:39:47.315 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:47.315 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:39:47.315 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:47.315 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:47.315 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-25 18:35:01.351486206 +0000 2024-03-25T19:39:47.316 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-25 18:29:12.405564708 +0000 2024-03-25T19:39:47.316 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-25 18:29:12.405564708 +0000 2024-03-25T19:39:47.316 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-25 18:29:12.405564708 +0000 2024-03-25T19:39:47.316 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:39:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-25T19:39:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-25T19:39:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000238649 s, 2.1 MB/s 2024-03-25T19:39:47.382 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:39:47.438 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-03-25T19:39:47.493 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:39:47.493 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:47.494 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T19:39:47.494 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:47.494 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:47.494 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-25 18:35:01.351486206 +0000 2024-03-25T19:39:47.494 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-25 18:29:12.744556361 +0000 2024-03-25T19:39:47.494 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-25 18:29:12.744556361 +0000 2024-03-25T19:39:47.494 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-25 18:29:12.744556361 +0000 2024-03-25T19:39:47.494 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:39:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-25T19:39:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-25T19:39:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000206151 s, 2.5 MB/s 2024-03-25T19:39:47.560 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:39:47.615 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-03-25T19:39:47.670 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:39:47.670 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:47.670 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T19:39:47.671 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:47.671 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:47.671 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-25 18:35:01.351486206 +0000 2024-03-25T19:39:47.671 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-25 18:29:13.070548333 +0000 2024-03-25T19:39:47.671 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-25 18:29:13.070548333 +0000 2024-03-25T19:39:47.671 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-25 18:29:13.070548333 +0000 2024-03-25T19:39:47.671 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:39:47.735 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-25T19:39:47.736 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-25T19:39:47.736 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000200993 s, 2.5 MB/s 2024-03-25T19:39:47.737 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:39:47.792 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:47.792 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:39:47.792 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:39:47.809 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:39:47.809 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-03-25T19:39:47.865 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:39:47.866 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:47.866 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:39:47.866 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:47.866 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:47.866 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-25 18:35:53.111609723 +0000 2024-03-25T19:39:47.866 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-25 18:29:12.064256031 +0000 2024-03-25T19:39:47.866 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-25 18:29:12.064256031 +0000 2024-03-25T19:39:47.866 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-25 18:29:12.064256031 +0000 2024-03-25T19:39:47.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:39:47.934 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-25T19:39:47.934 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-25T19:39:47.934 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000194507 s, 2.6 MB/s 2024-03-25T19:39:47.935 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:39:47.992 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-03-25T19:39:48.048 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:39:48.048 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:48.048 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T19:39:48.048 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:48.048 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:48.049 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-25 18:35:53.111609723 +0000 2024-03-25T19:39:48.049 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-25 18:29:12.408248761 +0000 2024-03-25T19:39:48.049 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-25 18:29:12.408248761 +0000 2024-03-25T19:39:48.049 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-25 18:29:12.408248761 +0000 2024-03-25T19:39:48.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:39:48.114 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-25T19:39:48.114 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-25T19:39:48.115 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000189472 s, 2.7 MB/s 2024-03-25T19:39:48.115 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:39:48.172 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-03-25T19:39:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:39:48.227 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:48.227 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T19:39:48.228 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:48.228 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:48.228 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-25 18:35:53.111609723 +0000 2024-03-25T19:39:48.228 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-25 18:29:12.737241808 +0000 2024-03-25T19:39:48.228 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-25 18:29:12.737241808 +0000 2024-03-25T19:39:48.228 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-25 18:29:12.737241808 +0000 2024-03-25T19:39:48.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:39:48.294 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-25T19:39:48.294 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-25T19:39:48.294 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000211413 s, 2.4 MB/s 2024-03-25T19:39:48.295 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:39:48.351 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-03-25T19:39:48.406 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:39:48.407 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:48.407 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:39:48.407 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:48.407 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:48.407 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-25 18:35:53.112609702 +0000 2024-03-25T19:39:48.407 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-25 18:29:13.075234665 +0000 2024-03-25T19:39:48.407 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-25 18:29:13.075234665 +0000 2024-03-25T19:39:48.407 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-25 18:29:13.075234665 +0000 2024-03-25T19:39:48.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:39:48.472 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-25T19:39:48.472 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-25T19:39:48.472 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000208307 s, 2.5 MB/s 2024-03-25T19:39:48.473 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:39:48.529 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:48.529 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:39:48.529 INFO:tasks.ceph:Generating config... 2024-03-25T19:39:48.530 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:39:48.530 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:39:48.530 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:39:48.530 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:39:48.531 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:39:48.531 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:39:48.531 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:39:48.531 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:39:48.531 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:39:48.531 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:39:48.531 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:39:48.531 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:39:48.531 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:39:48.531 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:39:48.531 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:39:48.531 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:39:48.531 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:39:48.531 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:39:48.532 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:39:48.532 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:39:48.532 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:39:48.532 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:39:48.532 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:39:48.532 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:39:48.532 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:39:48.532 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:39:48.532 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:39:48.532 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:39:48.532 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:39:48.532 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:39:48.532 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:39:48.533 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:39:48.533 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:39:48.533 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:39:48.533 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:39:48.533 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:39:48.534 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:39:48.535 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:39:48.535 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:39:48.535 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:39:48.535 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:39:48.535 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:39:48.535 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:39:48.535 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:39:48.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:39:48.589 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:39:48.592 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:39:48.636 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:48.662 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.69:6789]'), ('mon.c', '[v1:172.21.15.69:6790]'), ('mon.b', '[v1:172.21.15.169:6789]')] 2024-03-25T19:39:48.662 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.69:6789],[v1:172.21.15.69:6790],[v1:172.21.15.169:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:39:48.663 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:48.663 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:39:48.717 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.69:6789]' --addv c '[v1:172.21.15.69:6790]' --addv b '[v1:172.21.15.169:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: generated fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = quincy 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 0 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:min_mon_release 17 (quincy) 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:election_strategy: 3 2024-03-25T19:39:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:39:48.802 INFO:teuthology.orchestra.run.smithi069.stdout:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:39:48.802 INFO:teuthology.orchestra.run.smithi069.stdout:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:39:48.802 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:39:48.803 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:39:48.817 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0b4416d3-973b-4b34-9bff-820f05cd8197... 2024-03-25T19:39:48.818 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:39:48.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:39:48.907 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-03-25T19:39:48.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-03-25T19:39:48.909 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-03-25T19:39:48.910 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "[v1:172.21.15.69:6789],[v1:172.21.15.69:6790],[v1:172.21.15.169:6789]" 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: mon client directed command retry = 5 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: mon election default strategy = 3 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: ms bind msgr2 = False 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: ms inject socket failures = 5000 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: ms type = async 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: osd heartbeat grace = 80 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:39:48.911 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.912 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = stupid 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore zero block detection = True 2024-03-25T19:39:48.913 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: osd class default list = * 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: osd class load list = * 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: osd client watch timeout = 120 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify cached snaps = True 2024-03-25T19:39:48.914 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify missing on start = True 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: osd fast shutdown = False 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: osd max markdown count = 1000 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock override recovery settings = True 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue = debug_random 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue cut off = debug_random 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: osd shutdown pgref assert = True 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-25T19:39:48.915 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:48.916 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon min osdmap epochs = 50 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd crush smoke test = False 2024-03-25T19:39:48.917 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune interval = 2 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune min = 15 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: mon scrub interval = 300 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on pool no app = False 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: paxos service trim min = 10 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: debug objecter = 20 2024-03-25T19:39:48.918 INFO:teuthology.orchestra.run.smithi069.stdout: debug rados = 20 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:39:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:39:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "[v1:172.21.15.69:6789],[v1:172.21.15.69:6790],[v1:172.21.15.169:6789]" 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: mon client directed command retry = 5 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: mon election default strategy = 3 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: ms bind msgr2 = False 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject socket failures = 5000 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: ms type = async 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: osd heartbeat grace = 80 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-25T19:39:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:39:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = stupid 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore zero block detection = True 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: debug monc = 20 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-25T19:39:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 10 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd class default list = * 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd class load list = * 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd client watch timeout = 120 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify cached snaps = True 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify missing on start = True 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd fast shutdown = False 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd max markdown count = 1000 2024-03-25T19:39:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock override recovery settings = True 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: osd op queue = debug_random 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: osd op queue cut off = debug_random 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: osd shutdown pgref assert = True 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.927 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:39:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon min osdmap epochs = 50 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd crush smoke test = False 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune interval = 2 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune min = 15 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:39:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: mon scrub interval = 300 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on pool no app = False 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: paxos service trim min = 10 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: debug monc = 20 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: debug objecter = 20 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: debug rados = 20 2024-03-25T19:39:48.930 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-25T19:39:48.931 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-25T19:39:48.931 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-25T19:39:48.932 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:39:48.932 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:39:48.987 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:39:48.987 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:48.987 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:39:49.001 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:49.001 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:39:49.056 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:39:49.056 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:49.056 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:39:49.056 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:49.135 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:39:49.135 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:49.190 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:39:49.190 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:39:49.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:39:49.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:49.231 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:39:49.231 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:49.287 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:39:49.287 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:39:49.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:39:49.341 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:39:49.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:39:49.410 INFO:teuthology.orchestra.run.smithi169.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:39:49.412 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:39:49.412 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:39:49.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:39:49.454 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:39:49.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:39:49.512 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:39:49.527 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:39:49.527 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:39:49.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:49.554 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:49.554 INFO:tasks.ceph:role: osd.0 2024-03-25T19:39:49.554 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:39:49.554 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:39:49.628 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:49.628 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:49.628 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:49.629 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:49.629 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:49.629 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:49.629 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:49.629 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:49.629 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:49.629 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:50.142 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-03-25T19:39:50.144 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-03-25T19:39:50.144 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:50.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:50.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:50.647 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:50.647 INFO:tasks.ceph:role: osd.1 2024-03-25T19:39:50.648 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:39:50.648 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:39:50.721 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:50.721 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:50.721 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:50.721 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:50.721 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:50.721 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:50.721 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:50.722 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:50.722 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:50.722 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:51.198 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-03-25T19:39:51.200 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-03-25T19:39:51.200 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:51.294 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:51.324 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:51.391 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:51.391 INFO:tasks.ceph:role: osd.2 2024-03-25T19:39:51.391 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:39:51.391 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:39:51.464 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:51.464 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:51.464 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:51.464 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:51.465 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:51.465 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:51.465 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:51.465 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:51.465 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:51.465 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:51.982 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-03-25T19:39:51.984 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-03-25T19:39:51.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:52.126 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:52.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:52.225 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:52.225 INFO:tasks.ceph:role: osd.3 2024-03-25T19:39:52.225 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:39:52.225 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:39:52.298 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:52.298 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:52.298 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:52.298 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:52.298 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:52.298 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:52.298 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:52.299 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:52.299 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:52.299 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:52.805 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-03-25T19:39:52.807 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-03-25T19:39:52.807 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:52.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:52.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:53.044+0000 7f281d81b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:39:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:53.044+0000 7f281d81b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:39:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:53.044+0000 7f281d81b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:53.044+0000 7f281d81b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:53.044+0000 7f281d81b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:53.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:53.044+0000 7f281d81b640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:39:55.891 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:55.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:56.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:56.015+0000 7fc80d013640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:39:56.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:56.016+0000 7fc80d013640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:39:56.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:56.016+0000 7fc80d013640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:56.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:56.016+0000 7fc80d013640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:56.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:56.016+0000 7fc80d013640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:56.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:56.016+0000 7fc80d013640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:39:58.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:58.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:58.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:58.978+0000 7fb80a011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:39:58.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:58.978+0000 7fb80a011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:39:58.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:58.978+0000 7fb80a011640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:58.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:58.978+0000 7fb80a011640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:58.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:58.978+0000 7fb80a011640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:58.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:39:58.979+0000 7fb80a011640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:40:01.836 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:01.865 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:01.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:01.953+0000 7f441dc15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:40:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:01.954+0000 7f441dc15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:40:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:01.954+0000 7f441dc15640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:01.954+0000 7f441dc15640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:01.954+0000 7f441dc15640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:01.954+0000 7f441dc15640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:40:04.790 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:04.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:04.855 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:04.855 INFO:tasks.ceph:role: osd.4 2024-03-25T19:40:04.855 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:40:04.855 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:40:04.933 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:04.934 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:05.450 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-25T19:40:05.452 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-25T19:40:05.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:05.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:05.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:05.689 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:05.690 INFO:tasks.ceph:role: osd.5 2024-03-25T19:40:05.690 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:40:05.690 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:40:05.766 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:05.766 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:05.766 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:05.766 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:05.766 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:05.766 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:05.767 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:05.767 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:05.767 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:05.767 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:06.225 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-25T19:40:06.228 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-25T19:40:06.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:06.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:06.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:06.416 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:06.416 INFO:tasks.ceph:role: osd.6 2024-03-25T19:40:06.416 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:40:06.416 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:40:06.490 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:06.490 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:06.490 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:06.491 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:06.491 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:06.491 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:06.491 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:06.491 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:06.491 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:06.491 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:06.974 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-25T19:40:06.976 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-25T19:40:06.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:07.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:07.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:07.200 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:07.200 INFO:tasks.ceph:role: osd.7 2024-03-25T19:40:07.200 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:40:07.200 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:40:07.274 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:07.274 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:07.275 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:07.275 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:07.275 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:07.275 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:07.275 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:07.275 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:07.275 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:07.275 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:07.769 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-25T19:40:07.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-25T19:40:07.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:07.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:07.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:08.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:08.048+0000 7fc304815640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:40:08.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:08.048+0000 7fc304815640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:40:08.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:08.048+0000 7fc304815640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:08.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:08.048+0000 7fc304815640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:08.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:08.048+0000 7fc304815640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:08.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:08.049+0000 7fc304815640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:40:10.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:10.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:11.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:11.022+0000 7fcab64f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:40:11.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:11.022+0000 7fcab64f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:40:11.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:11.022+0000 7fcab64f9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:11.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:11.023+0000 7fcab64f9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:11.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:11.023+0000 7fcab64f9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:11.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:11.023+0000 7fcab64f9640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:40:13.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:13.896 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:13.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:13.987+0000 7f8218024640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:40:13.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:13.988+0000 7f8218024640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:40:13.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:13.988+0000 7f8218024640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:13.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:13.988+0000 7f8218024640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:13.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:13.988+0000 7f8218024640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:13.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:13.988+0000 7f8218024640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:40:16.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:16.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:16.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:16.954+0000 7f42e4e15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:40:16.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:16.954+0000 7f42e4e15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:40:16.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:16.954+0000 7f42e4e15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:16.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:16.954+0000 7f42e4e15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:16.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:16.954+0000 7f42e4e15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:16.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:16.955+0000 7f42e4e15640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:40:19.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:19.831 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:40:19.831 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:19.831 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:40:19.863 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:19.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:40:19.933 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:19.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:40:20.003 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:20.003 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:40:20.071 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:20.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:40:20.140 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:20.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:40:20.172 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:20.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:40:20.243 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:20.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:40:20.312 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:20.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:40:20.381 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:20.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:40:20.449 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:20.449 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:40:20.467 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:20.467 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:40:20.505 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:40:20.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:40:20.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:40:20.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2024-03-25T19:40:20.548 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAF0wFmbFF1FBAAukMCeMILrgZcg1ixe+37Sw== 2024-03-25T19:40:20.548 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2024-03-25T19:40:20.548 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAJ0wFmkymlAhAA/iknmJA/J2qYokFnnD14xg== 2024-03-25T19:40:20.548 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2024-03-25T19:40:20.548 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAM0wFmd1bzABAAUUbHydokAxDpABTafXJEPg== 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAO0wFmd3pSOhAANV79zFlg6aXoc2CbiW1k7g== 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAR0wFmJFPrOBAAI2O14dsO1KnkcjEtBLSdaw== 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAF0wFm1QGIGBAAwof5n7bhTBIYKPaDsi+/uA== 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAY0wFm4WfwAhAAzBvVuZJJiYKLXzhOOyFwmg== 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAb0wFm36RnARAAOGx02aW7pbdrILLbyszs/A== 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAd0wFm2RnvOhAA1JEcSxL4tEQi5ug39MC1xw== 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAg0wFmp6XxOBAAN6RSLXgN/D0SYbXFyxME+g== 2024-03-25T19:40:20.549 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-03-25T19:40:20.550 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAF0wFmTW4wGxAA7ZaiJxYeaxbQ7YVvZAxcRQ== 2024-03-25T19:40:20.550 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2024-03-25T19:40:20.550 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAF0wFmvOGgHhAAdsqltvjw/Tk40M6Jc2yhKQ== 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.y] 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAF0wFmbFF1FBAAukMCeMILrgZcg1ixe+37Sw== 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.0] 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAJ0wFmkymlAhAA/iknmJA/J2qYokFnnD14xg== 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.1] 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAM0wFmd1bzABAAUUbHydokAxDpABTafXJEPg== 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.2] 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAO0wFmd3pSOhAANV79zFlg6aXoc2CbiW1k7g== 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.3] 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAR0wFmJFPrOBAAI2O14dsO1KnkcjEtBLSdaw== 2024-03-25T19:40:20.571 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.x] 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAF0wFm1QGIGBAAwof5n7bhTBIYKPaDsi+/uA== 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.4] 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAY0wFm4WfwAhAAzBvVuZJJiYKLXzhOOyFwmg== 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.5] 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAb0wFm36RnARAAOGx02aW7pbdrILLbyszs/A== 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.6] 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAd0wFm2RnvOhAA1JEcSxL4tEQi5ug39MC1xw== 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.7] 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAg0wFmp6XxOBAAN6RSLXgN/D0SYbXFyxME+g== 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAF0wFmTW4wGxAA7ZaiJxYeaxbQ7YVvZAxcRQ== 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout:[client.1] 2024-03-25T19:40:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAF0wFmvOGgHhAAdsqltvjw/Tk40M6Jc2yhKQ== 2024-03-25T19:40:20.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:20.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:20.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:20.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:20.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:20.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:20.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:20.775 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:20.826 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:20.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:20.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:20.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:21.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:21.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:21.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:21.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:21.114 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:21.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:21.167 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:21.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:21.221 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:21.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:21.278 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:21.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:21.335 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:40:21.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:40:21.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:21.457 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:40:21.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:40:21.550 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:21.645 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:40:21.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:40:21.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:21.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:40:21.822 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:21.824 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:21.877 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:40:21.877 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:21.877 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:40:21.877 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:40:21.880 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:40:21.880 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:21.880 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:40:21.880 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:40:21.882 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:40:21.882 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:21.882 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:40:21.882 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:40:21.920 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:40:21.920 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:40:21.920 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:40:21.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:40:21.923 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:40:21.923 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:40:21.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:40:21.925 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:40:21.925 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:40:21.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:40:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.025+0000 7f727deba640 10 monclient: get_monmap_and_config 2024-03-25T19:40:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.025+0000 7f727deba640 10 monclient: build_initial_monmap 2024-03-25T19:40:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.025+0000 7f727deba640 10 monclient: monmap: 2024-03-25T19:40:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-03-25T19:40:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:40:22.026668+0000 2024-03-25T19:40:22.025 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:40:22.026668+0000 2024-03-25T19:40:22.026 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:22.026 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:22.026 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:22.026 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:22.026 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:22.026 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:22.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 1 Processor -- start 2024-03-25T19:40:22.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 1 -- start start 2024-03-25T19:40:22.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 10 monclient: init 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 10 monclient(hunting): picked mon.noname-c con 0x7f7278070210 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 10 monclient(hunting): picked mon.noname-b con 0x7f72780b06d0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 10 monclient(hunting): picked mon.noname-a con 0x7f72780705f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780b0b80 con 0x7f72780705f0 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780b0430 con 0x7f7278070210 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7278071670 con 0x7f72780b06d0 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f727deba640 10 monclient(hunting): authenticate will time out at 4834.295473s 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f7277fff640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f727814b960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f72777fe640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f7278140930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.026+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f72780afd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.227+0000 7f72777fe640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f7278140930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.227+0000 7f7277fff640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f727814b960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.227+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f72780afd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:22.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.628+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f72780afd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:22.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.628+0000 7f7277fff640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f727814b960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:22.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:22.628+0000 7f72777fe640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f7278140930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:23.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:23.429+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f72780afd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:23.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:23.429+0000 7f7277fff640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f727814b960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:23.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:23.429+0000 7f72777fe640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f7278140930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:25.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 10 monclient(hunting): tick 2024-03-25T19:40:25.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:25.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:25.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:25.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f72780afd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:25.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f7278140930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:25.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f727814b960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:25.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-a con 0x7f72780705f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-b con 0x7f72780b06d0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-c con 0x7f7278070210 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7278070a60 con 0x7f72780705f0 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780b0b80 con 0x7f7278070210 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7278071670 con 0x7f72780b06d0 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72757fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.026+0000 7f72777fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f72640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:25.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.027+0000 7f7277fff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:25.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.228+0000 7f72777fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f72640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:25.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.228+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:25.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.228+0000 7f7277fff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:25.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.629+0000 7f72777fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f72640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:25.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.629+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:25.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:25.629+0000 7f7277fff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:26.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:26.430+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:26.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:26.430+0000 7f72777fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f72640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:26.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:26.430+0000 7f7277fff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:28.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 10 monclient(hunting): tick 2024-03-25T19:40:28.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:28.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:28.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:28.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:28.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:28.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f72640056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-a con 0x7f72780705f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-b con 0x7f72780b06d0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-c con 0x7f7278070210 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264011000 con 0x7f72780705f0 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7278070a60 con 0x7f7278070210 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780b0b80 con 0x7f72780b06d0 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72757fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f72777fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f726400ff30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:28.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.027+0000 7f7277fff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:28.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.228+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:28.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.228+0000 7f72777fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f726400ff30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:28.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.228+0000 7f7277fff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:28.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.629+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:28.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.629+0000 7f72777fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f726400ff30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:28.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:28.629+0000 7f7277fff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:28.958 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:00:06.912 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:29.216 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:00:07.180 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:29.218 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:00:07.182 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:29.266 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:00:07.220 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:29.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:29.430+0000 7f72777fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f726400ff30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-03-25T19:40:29.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:29.430+0000 7f7276ffd640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6790/0 2024-03-25T19:40:29.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:29.430+0000 7f7277fff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T19:40:29.538 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:00:07.502 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:29.539 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:00:07.503 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 10 monclient(hunting): tick 2024-03-25T19:40:31.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:31.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:31.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:31.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 1 -- >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:31.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f7278070210 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:31.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f726400ff30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-c con 0x7f72640056c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-a con 0x7f72780705f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-b con 0x7f72780b06d0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7278071670 con 0x7f72780705f0 2024-03-25T19:40:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264011000 con 0x7f72640056c0 2024-03-25T19:40:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7278070a60 con 0x7f72780b06d0 2024-03-25T19:40:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.027+0000 7f72757fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:31.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.058+0000 7f72777fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f72640056c0 0x7f72640106c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:39418/0 (socket says 172.21.15.69:39418) 2024-03-25T19:40:31.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:31.058+0000 7f72777fe640 1 -- 172.21.15.69:0/2138697321 learned_addr learned my addr 172.21.15.69:0/2138697321 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:40:34.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 10 monclient(hunting): tick 2024-03-25T19:40:34.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:34.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.169:6789/0 conn(0x7f72640056c0 legacy=0x7f72640106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6789/0 conn(0x7f72780705f0 legacy=0x7f726400ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-c con 0x7f726400ce30 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-a con 0x7f72640106c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-b con 0x7f72780b06d0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780b0b80 con 0x7f72640106c0 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264010af0 con 0x7f726400ce30 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726400d980 con 0x7f72780b06d0 2024-03-25T19:40:34.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:34.028+0000 7f72757fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 10 monclient(hunting): tick 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6790/0 conn(0x7f72780b06d0 legacy=0x7f7264008e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.169:6789/0 conn(0x7f726400ce30 legacy=0x7f726400c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6789/0 conn(0x7f72640106c0 legacy=0x7f726400d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-a con 0x7f726400ff10 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-b con 0x7f7264006510 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-c con 0x7f726400ce30 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:37.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726400db10 con 0x7f726400ff10 2024-03-25T19:40:37.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264004650 con 0x7f726400ce30 2024-03-25T19:40:37.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72640047e0 con 0x7f7264006510 2024-03-25T19:40:37.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:37.028+0000 7f72757fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:40.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 10 monclient(hunting): tick 2024-03-25T19:40:40.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:40.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:40.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6790/0 conn(0x7f7264006510 legacy=0x7f72640162b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.169:6789/0 conn(0x7f726400ce30 legacy=0x7f726400d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6789/0 conn(0x7f726400ff10 legacy=0x7f7264005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-a con 0x7f726400c8a0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-b con 0x7f7264006510 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-c con 0x7f726400ce30 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726400c710 con 0x7f726400c8a0 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264005080 con 0x7f726400ce30 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72640052b0 con 0x7f7264006510 2024-03-25T19:40:40.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:40.029+0000 7f72757fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 10 monclient(hunting): tick 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6790/0 conn(0x7f7264006510 legacy=0x7f7264005de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.169:6789/0 conn(0x7f726400ce30 legacy=0x7f726400d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6789/0 conn(0x7f726400c8a0 legacy=0x7f7264004970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-a con 0x7f7264008ae0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-c con 0x7f726400ce30 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 10 monclient(hunting): picked mon.noname-b con 0x7f7264006510 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264008930 con 0x7f7264008ae0 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264008f20 con 0x7f726400ce30 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264009150 con 0x7f7264006510 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:43.028+0000 7f72757fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:45.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.426+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516077662 0 0) 0x7f7264008930 con 0x7f7264008ae0 2024-03-25T19:40:45.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.426+0000 7f72767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:45.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.426+0000 7f72767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:45.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.426+0000 7f72767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:45.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.426+0000 7f72767fc640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:40:45.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.426+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7260003520 con 0x7f7264008ae0 2024-03-25T19:40:45.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.448+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460016672 0 0) 0x7f7264008f20 con 0x7f726400ce30 2024-03-25T19:40:45.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.448+0000 7f72767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:45.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.448+0000 7f72767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:45.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.448+0000 7f72767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:45.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.448+0000 7f72767fc640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:40:45.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.448+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7264008930 con 0x7f726400ce30 2024-03-25T19:40:45.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.455+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200365564 0 0) 0x7f7264009150 con 0x7f7264006510 2024-03-25T19:40:45.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.455+0000 7f72767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:45.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.455+0000 7f72767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:45.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.455+0000 7f72767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:45.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.455+0000 7f72767fc640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:40:45.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.455+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7264008f20 con 0x7f7264006510 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377295969 0 0) 0x7f7260003520 con 0x7f7264008ae0 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6790/0 conn(0x7f7264006510 legacy=0x7f7264005de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.169:6789/0 conn(0x7f726400ce30 legacy=0x7f726400d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 1 monclient: found mon.noname-a 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264009380 con 0x7f7264008ae0 2024-03-25T19:40:45.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7268003fb0 con 0x7f7264008ae0 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f72767fc640 10 monclient: dump: 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:45.480 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:45.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f727deba640 5 monclient: authenticate success, global_id 4101 2024-03-25T19:40:45.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.479+0000 7f727deba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7268004110 con 0x7f7264008ae0 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f72767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f72767fc640 1 -- 172.21.15.69:0/2138697321 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7268004520 con 0x7f7264008ae0 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f727deba640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f72767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f727deba640 4 set_mon_vals no callback set 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f72767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f72767fc640 10 monclient: dump: 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:45.527 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:45.527 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:45.527 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:45.527 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:45.527 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:45.527 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:45.527 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f727deba640 10 monclient: shutdown 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f727deba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f727deba640 1 -- 172.21.15.69:0/2138697321 >> v1:172.21.15.69:6789/0 conn(0x7f7264008ae0 legacy=0x7f7264007920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f727deba640 1 -- 172.21.15.69:0/2138697321 shutdown_connections 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.526+0000 7f727deba640 1 -- 172.21.15.69:0/2138697321 >> 172.21.15.69:0/2138697321 conn(0x7f72780b5cd0 msgr2=0x7f72780b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.527+0000 7f727deba640 1 -- 172.21.15.69:0/2138697321 shutdown_connections 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.527+0000 7f727deba640 1 -- 172.21.15.69:0/2138697321 wait complete. 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.527+0000 7f727deba640 10 monclient: build_initial_monmap 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.527+0000 7f727deba640 10 monclient: monmap: 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.527+0000 7f727deba640 1 librados: starting msgr at 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.527+0000 7f727deba640 1 librados: starting objecter 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 1 Processor -- start 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 1 -- start start 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 1 librados: setting wanted keys 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 1 librados: calling monclient init 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 10 monclient: init 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 10 monclient(hunting): picked mon.noname-c con 0x7f7264006510 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 10 monclient(hunting): picked mon.noname-a con 0x7f7264008ae0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 10 monclient(hunting): picked mon.noname-b con 0x7f726400ce30 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7278136870 con 0x7f7264008ae0 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72781db670 con 0x7f726400ce30 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72781db340 con 0x7f7264006510 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.528+0000 7f727deba640 10 monclient(hunting): authenticate will time out at 4857.797432s 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.529+0000 7f72777fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f7264006510 0x7f72781dac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:51214/0 (socket says 172.21.15.69:51214) 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.529+0000 7f72777fe640 1 -- 172.21.15.69:0/3866863774 learned_addr learned my addr 172.21.15.69:0/3866863774 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.529+0000 7f725ffff640 10 client.?.objecter ms_handle_connect 0x7f7264006510 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.529+0000 7f725ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:45.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.535+0000 7f725ffff640 10 client.?.objecter ms_handle_connect 0x7f7264008ae0 2024-03-25T19:40:45.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.535+0000 7f725ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.549+0000 7f725ffff640 10 client.?.objecter ms_handle_connect 0x7f726400ce30 2024-03-25T19:40:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.549+0000 7f725ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.551+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519778400 0 0) 0x7f7278136870 con 0x7f7264008ae0 2024-03-25T19:40:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.551+0000 7f725ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.551+0000 7f725ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.551+0000 7f725ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.551+0000 7f725ffff640 10 monclient(hunting): my global_id is 4110 2024-03-25T19:40:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.551+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7264009150 con 0x7f7264008ae0 2024-03-25T19:40:45.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.624+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263594605 0 0) 0x7f72781db670 con 0x7f726400ce30 2024-03-25T19:40:45.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.624+0000 7f725ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:45.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.624+0000 7f725ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:45.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.624+0000 7f725ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:45.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.624+0000 7f725ffff640 10 monclient(hunting): my global_id is 4108 2024-03-25T19:40:45.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.624+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7278136870 con 0x7f726400ce30 2024-03-25T19:40:45.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.628+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454394421 0 0) 0x7f72781db340 con 0x7f7264006510 2024-03-25T19:40:45.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.628+0000 7f725ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:45.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.628+0000 7f725ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:45.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.628+0000 7f725ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:45.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.628+0000 7f725ffff640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:40:45.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.628+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72781db670 con 0x7f7264006510 2024-03-25T19:40:45.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.665+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1516638979 0 0) 0x7f7264009150 con 0x7f7264008ae0 2024-03-25T19:40:45.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.665+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72781db340 con 0x7f7264008ae0 2024-03-25T19:40:45.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.665+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f726c004320 con 0x7f7264008ae0 2024-03-25T19:40:45.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.665+0000 7f725ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:45.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.665+0000 7f725ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:45.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.665+0000 7f725ffff640 10 monclient(hunting): dump: 2024-03-25T19:40:45.665 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:45.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.665+0000 7f725ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:45.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3502871749 0 0) 0x7f72781db340 con 0x7f7264008ae0 2024-03-25T19:40:45.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 >> v1:172.21.15.69:6790/0 conn(0x7f7264006510 legacy=0x7f72781dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:45.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 >> v1:172.21.15.169:6789/0 conn(0x7f726400ce30 legacy=0x7f72780aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:45.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 1 monclient: found mon.a 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7278135dd0 con 0x7f7264008ae0 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f725ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 5 monclient: authenticate success, global_id 4110 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 10 monclient: _renew_subs 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7278136020 con 0x7f7264008ae0 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 10 monclient: _renew_subs 2024-03-25T19:40:45.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:45.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72781359d0 con 0x7f7264008ae0 2024-03-25T19:40:45.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.714+0000 7f727deba640 1 librados: init done 2024-03-25T19:40:45.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.715+0000 7f727deba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:45.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.715+0000 7f727deba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:45.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.715+0000 7f727deba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:45.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.715+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72781359d0 con 0x7f7264008ae0 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f726c004730 con 0x7f7264008ae0 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f726c004ac0 con 0x7f7264008ae0 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 10 monclient: dump: 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:45.948 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f726c005070 con 0x7f7264008ae0 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1878904067 0 0) 0x7f726c0064e0 con 0x7f7264008ae0 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 10 client.4110.objecter ms_dispatch 0x7f7278000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f7275ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f726c006900 con 0x7f7264008ae0 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:45.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:45.947+0000 7f725ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:46.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.049+0000 7f727deba640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:46.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.049+0000 7f727deba640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:46.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.049+0000 7f727deba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:46.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.049+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f72781359d0 con 0x7f7264008ae0 2024-03-25T19:40:46.106 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:46.106+0000 7f63ebc71040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:46.139 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:46.139+0000 7f1af0443040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:46.163 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:46.163+0000 7f63ebc71040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:46.192 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:46.192+0000 7f1af0443040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:46.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.285+0000 7f725ffff640 1 -- 172.21.15.69:0/3866863774 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f726c005400 con 0x7f7264008ae0 2024-03-25T19:40:46.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.285+0000 7f725ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:46.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.285+0000 7f725ffff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:40:46.286 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2024-03-25T19:40:46.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.286+0000 7f727deba640 10 librados: watch_flush enter 2024-03-25T19:40:46.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.286+0000 7f727deba640 10 librados: watch_flush exit 2024-03-25T19:40:46.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.286+0000 7f727deba640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T19:40:46.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.286+0000 7f727deba640 10 client.4110.objecter successfully canceled tick 2024-03-25T19:40:46.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.287+0000 7f727deba640 10 monclient: shutdown 2024-03-25T19:40:46.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.287+0000 7f727deba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:46.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.287+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 >> v1:172.21.15.69:6789/0 conn(0x7f7264008ae0 legacy=0x7f72781350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.287+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 shutdown_connections 2024-03-25T19:40:46.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.287+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 >> 172.21.15.69:0/3866863774 conn(0x7f72780b5cd0 msgr2=0x7f72780b53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:46.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.287+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 shutdown_connections 2024-03-25T19:40:46.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.287+0000 7f727deba640 1 -- 172.21.15.69:0/3866863774 wait complete. 2024-03-25T19:40:46.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:46.287+0000 7f727deba640 1 librados: shutdown 2024-03-25T19:40:46.298 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:40:46.298 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:40:46.298 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:46.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:40:46.325 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:46.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:40:46.393 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:46.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:40:46.462 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-25T19:40:46.462 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:40:46.528 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:46.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:40:46.556 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:46.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:40:46.623 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:46.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:40:46.690 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T19:40:46.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:40:46.738 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:46.739+0000 7f1af0443040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:46.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 7c9ae774-73d1-442a-8d13-58702d808d34 0 2024-03-25T19:40:46.766 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:46.766+0000 7f63ebc71040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:46.795 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:46.795+0000 7f1af0443040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:46.820 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:46.820+0000 7f63ebc71040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:46.843 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:46.843+0000 7f1af0443040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:46.864 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:46.864+0000 7f63ebc71040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:46.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.894+0000 7fca8bb32640 10 monclient: get_monmap_and_config 2024-03-25T19:40:46.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.894+0000 7fca8bb32640 10 monclient: build_initial_monmap 2024-03-25T19:40:46.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.894+0000 7fca8bb32640 10 monclient: monmap: 2024-03-25T19:40:46.894 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:40:46.895784+0000 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:40:46.895784+0000 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.895+0000 7fca8bb32640 1 Processor -- start 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.895+0000 7fca8bb32640 1 -- start start 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.895+0000 7fca8bb32640 10 monclient: init 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 10 monclient(hunting): picked mon.noname-b con 0x7fca840b4700 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 10 monclient(hunting): picked mon.noname-c con 0x7fca84142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 10 monclient(hunting): picked mon.noname-a con 0x7fca840a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca840a9d60 con 0x7fca840a96b0 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca8413ff10 con 0x7fca84142a20 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca840aa1f0 con 0x7fca840b4700 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8bb32640 10 monclient(hunting): authenticate will time out at 4853.196782s 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca890cd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fca84142a20 0x7fca8413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:46914/0 (socket says 172.21.15.169:46914) 2024-03-25T19:40:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca8a0cf640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fca840a96b0 0x7fca8413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.169:42776/0 (socket says 172.21.15.169:42776) 2024-03-25T19:40:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.896+0000 7fca890cd640 1 -- 172.21.15.169:0/1842494913 learned_addr learned my addr 172.21.15.169:0/1842494913 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.898+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153735993 0 0) 0x7fca8413ff10 con 0x7fca84142a20 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.898+0000 7fca888cc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.898+0000 7fca888cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.898+0000 7fca888cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.898+0000 7fca888cc640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.898+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca70003520 con 0x7fca84142a20 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395378699 0 0) 0x7fca840a9d60 con 0x7fca840a96b0 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca8413ff10 con 0x7fca840a96b0 2024-03-25T19:40:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392656021 0 0) 0x7fca840aa1f0 con 0x7fca840b4700 2024-03-25T19:40:46.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:40:46.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.899+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca840a9d60 con 0x7fca840b4700 2024-03-25T19:40:46.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3502086858 0 0) 0x7fca70003520 con 0x7fca84142a20 2024-03-25T19:40:46.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 >> v1:172.21.15.69:6790/0 conn(0x7fca840b4700 legacy=0x7fca84142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 >> v1:172.21.15.69:6789/0 conn(0x7fca840a96b0 legacy=0x7fca8413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:46.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 1 monclient: found mon.noname-c 2024-03-25T19:40:46.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:46.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:40:46.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca840b4ae0 con 0x7fca84142a20 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 10 monclient: _finish_auth 0 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca8bb32640 5 monclient: authenticate success, global_id 4114 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fca80001690 con 0x7fca84142a20 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca8bb32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca888cc640 10 monclient: dump: 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:46.903 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:46.904 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:46.904 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:46.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.902+0000 7fca8bb32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:46.908 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:46.908+0000 7f1af0443040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:46.919 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:46.920+0000 7f63ebc71040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.928+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca800017f0 con 0x7fca84142a20 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.928+0000 7fca888cc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.928+0000 7fca888cc640 1 -- 172.21.15.169:0/1842494913 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fca80001c00 con 0x7fca84142a20 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.928+0000 7fca888cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.928+0000 7fca888cc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.928+0000 7fca888cc640 10 monclient: dump: 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:46.928 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:46.929 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.929 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:46.929 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:46.929 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:46.929 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:46.929 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:46.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.928+0000 7fca8bb32640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:46.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.928+0000 7fca8bb32640 4 set_mon_vals no callback set 2024-03-25T19:40:46.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.929+0000 7fca8bb32640 10 monclient: shutdown 2024-03-25T19:40:46.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.929+0000 7fca8bb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:46.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.929+0000 7fca8bb32640 1 -- 172.21.15.169:0/1842494913 >> v1:172.21.15.169:6789/0 conn(0x7fca84142a20 legacy=0x7fca8413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.929+0000 7fca8bb32640 1 -- 172.21.15.169:0/1842494913 shutdown_connections 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.929+0000 7fca8bb32640 1 -- 172.21.15.169:0/1842494913 >> 172.21.15.169:0/1842494913 conn(0x7fca840668e0 msgr2=0x7fca84066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.929+0000 7fca8bb32640 1 -- 172.21.15.169:0/1842494913 shutdown_connections 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.929+0000 7fca8bb32640 1 -- 172.21.15.169:0/1842494913 wait complete. 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.930+0000 7fca8bb32640 10 monclient: build_initial_monmap 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.930+0000 7fca8bb32640 10 monclient: monmap: 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:46.931 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:46.932 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:46.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.931+0000 7fca8bb32640 1 librados: starting msgr at 2024-03-25T19:40:46.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.931+0000 7fca8bb32640 1 librados: starting objecter 2024-03-25T19:40:46.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.931+0000 7fca8bb32640 1 Processor -- start 2024-03-25T19:40:46.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.931+0000 7fca8bb32640 1 -- start start 2024-03-25T19:40:46.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.931+0000 7fca8bb32640 1 librados: setting wanted keys 2024-03-25T19:40:46.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.931+0000 7fca8bb32640 1 librados: calling monclient init 2024-03-25T19:40:46.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.931+0000 7fca8bb32640 10 monclient: init 2024-03-25T19:40:46.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:46.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 10 monclient(hunting): picked mon.noname-c con 0x7fca840a96b0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 10 monclient(hunting): picked mon.noname-a con 0x7fca840b4700 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 10 monclient(hunting): picked mon.noname-b con 0x7fca84142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca841d7540 con 0x7fca840b4700 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca8413af50 con 0x7fca84142a20 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca841d83c0 con 0x7fca840a96b0 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8bb32640 10 monclient(hunting): authenticate will time out at 4853.232933s 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8a0cf640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fca84142a20 0x7fca840ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:46922/0 (socket says 172.21.15.169:46922) 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.932+0000 7fca8a0cf640 1 -- 172.21.15.169:0/1618078497 learned_addr learned my addr 172.21.15.169:0/1618078497 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.933+0000 7fca7a7fc640 10 client.?.objecter ms_handle_connect 0x7fca84142a20 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.933+0000 7fca7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.933+0000 7fca7a7fc640 10 client.?.objecter ms_handle_connect 0x7fca840b4700 2024-03-25T19:40:46.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.933+0000 7fca7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568731292 0 0) 0x7fca841d7540 con 0x7fca840b4700 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca68004eb0 con 0x7fca840b4700 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783685901 0 0) 0x7fca8413af50 con 0x7fca84142a20 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:40:46.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.935+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca841d7540 con 0x7fca84142a20 2024-03-25T19:40:46.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.939+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 758607054 0 0) 0x7fca68004eb0 con 0x7fca840b4700 2024-03-25T19:40:46.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.939+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca8413af50 con 0x7fca840b4700 2024-03-25T19:40:46.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.940+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fca80001b80 con 0x7fca840b4700 2024-03-25T19:40:46.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.940+0000 7fca7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.940+0000 7fca7a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.940+0000 7fca7a7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:46.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.940+0000 7fca7a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:46.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2118214145 0 0) 0x7fca8413af50 con 0x7fca840b4700 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 >> v1:172.21.15.69:6790/0 conn(0x7fca840a96b0 legacy=0x7fca841d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 >> v1:172.21.15.169:6789/0 conn(0x7fca84142a20 legacy=0x7fca840ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 1 monclient: found mon.a 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca841d7790 con 0x7fca840b4700 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 10 monclient: _renew_subs 2024-03-25T19:40:46.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:46.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca840ab930 con 0x7fca840b4700 2024-03-25T19:40:46.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:46.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 10 monclient: _renew_subs 2024-03-25T19:40:46.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:46.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca841d5070 con 0x7fca840b4700 2024-03-25T19:40:46.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 1 librados: init done 2024-03-25T19:40:46.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.944+0000 7fca8bb32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.945+0000 7fca8bb32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.945+0000 7fca8bb32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:46.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.945+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca841d7980 con 0x7fca840b4700 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.947+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca80004a90 con 0x7fca840b4700 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.947+0000 7fca7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.947+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fca80004da0 con 0x7fca840b4700 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.947+0000 7fca7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.947+0000 7fca7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.947+0000 7fca7a7fc640 10 monclient: dump: 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:46.948 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:46.947+0000 7fca7bfff640 4 set_mon_vals no callback set 2024-03-25T19:40:46.962 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:46.963+0000 7f1af0443040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:46.965 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:46.965+0000 7f63ebc71040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:47.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.044+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fca80005290 con 0x7fca840b4700 2024-03-25T19:40:47.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.044+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 240749205 0 0) 0x7fca800066c0 con 0x7fca840b4700 2024-03-25T19:40:47.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.044+0000 7fca7a7fc640 10 client.4119.objecter ms_dispatch 0x7fca84000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:40:47.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.044+0000 7fca7a7fc640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:40:47.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.044+0000 7fca7a7fc640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:40:47.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.044+0000 7fca7a7fc640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:40:47.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.046+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fca80006ab0 con 0x7fca840b4700 2024-03-25T19:40:47.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.046+0000 7fca7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.046+0000 7fca7a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:47.079 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:47.080+0000 7f1af0443040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:47.085 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:47.085+0000 7f63ebc71040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:47.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.118+0000 7fca8bb32640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7c9ae774-73d1-442a-8d13-58702d808d34", "id": 0}] 2024-03-25T19:40:47.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.118+0000 7fca8bb32640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7c9ae774-73d1-442a-8d13-58702d808d34", "id": 0}] 2024-03-25T19:40:47.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.118+0000 7fca8bb32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:47.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.118+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7c9ae774-73d1-442a-8d13-58702d808d34", "id": 0} v 0) v1 -- 0x7fca8404f730 con 0x7fca840b4700 2024-03-25T19:40:47.150 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:47.151+0000 7f1af0443040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:47.156 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:47.157+0000 7f63ebc71040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:47.200 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:47.200+0000 7f1af0443040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:47.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.204+0000 7fca7a7fc640 1 -- 172.21.15.169:0/1618078497 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7c9ae774-73d1-442a-8d13-58702d808d34", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3968298931 0 732707515) 0x7fca80005540 con 0x7fca840b4700 2024-03-25T19:40:47.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.204+0000 7fca7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7c9ae774-73d1-442a-8d13-58702d808d34", "id": 0}] 2024-03-25T19:40:47.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.204+0000 7fca7a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:47.204 INFO:teuthology.orchestra.run.smithi169.stdout:0 2024-03-25T19:40:47.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 10 librados: watch_flush enter 2024-03-25T19:40:47.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 10 librados: watch_flush exit 2024-03-25T19:40:47.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 10 monclient: shutdown 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 >> v1:172.21.15.69:6789/0 conn(0x7fca840b4700 legacy=0x7fca841d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 shutdown_connections 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 >> 172.21.15.169:0/1618078497 conn(0x7fca840668e0 msgr2=0x7fca8413c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 shutdown_connections 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 1 -- 172.21.15.169:0/1618078497 wait complete. 2024-03-25T19:40:47.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.205+0000 7fca8bb32640 1 librados: shutdown 2024-03-25T19:40:47.207 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:47.206+0000 7f63ebc71040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:47.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 68679d85-c21b-44b4-9f57-a4af14c984d1 1 2024-03-25T19:40:47.263 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:47.263+0000 7f63ebc71040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:47.283 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:47.284+0000 7f1af0443040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:47.312 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:47.312+0000 7f63ebc71040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:47.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.316+0000 7f98653f6640 10 monclient: get_monmap_and_config 2024-03-25T19:40:47.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.316+0000 7f98653f6640 10 monclient: build_initial_monmap 2024-03-25T19:40:47.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.316+0000 7f98653f6640 10 monclient: monmap: 2024-03-25T19:40:47.315 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:40:47.317019+0000 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:40:47.317019+0000 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:47.316 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.316+0000 7f98653f6640 1 Processor -- start 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.316+0000 7f98653f6640 1 -- start start 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.316+0000 7f98653f6640 10 monclient: init 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 10 monclient(hunting): picked mon.noname-b con 0x7f98600b4700 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 10 monclient(hunting): picked mon.noname-c con 0x7f9860142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 10 monclient(hunting): picked mon.noname-a con 0x7f98600a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98600a9d60 con 0x7f98600a96b0 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986013ff10 con 0x7f9860142a20 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98600aa1f0 con 0x7f98600b4700 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f98653f6640 10 monclient(hunting): authenticate will time out at 4853.617867s 2024-03-25T19:40:47.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f985f7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f98600a96b0 0x7f986013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.169:42798/0 (socket says 172.21.15.169:42798) 2024-03-25T19:40:47.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.317+0000 7f985f7fe640 1 -- 172.21.15.169:0/511619079 learned_addr learned my addr 172.21.15.169:0/511619079 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407213086 0 0) 0x7f98600a9d60 con 0x7f98600a96b0 2024-03-25T19:40:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:40:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9848003520 con 0x7f98600a96b0 2024-03-25T19:40:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696607780 0 0) 0x7f98600aa1f0 con 0x7f98600b4700 2024-03-25T19:40:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98600a9d60 con 0x7f98600b4700 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980982131 0 0) 0x7f986013ff10 con 0x7f9860142a20 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:40:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.320+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98600aa1f0 con 0x7f9860142a20 2024-03-25T19:40:47.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950321114 0 0) 0x7f9848003520 con 0x7f98600a96b0 2024-03-25T19:40:47.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 >> v1:172.21.15.69:6790/0 conn(0x7f98600b4700 legacy=0x7f9860142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 >> v1:172.21.15.169:6789/0 conn(0x7f9860142a20 legacy=0x7f986013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:47.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 1 monclient: found mon.noname-a 2024-03-25T19:40:47.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:47.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:40:47.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98600b4ae0 con 0x7f98600a96b0 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9858001a10 con 0x7f98600a96b0 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f985dffb640 10 monclient: dump: 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.324 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.325 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.325 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.325 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f98653f6640 5 monclient: authenticate success, global_id 4122 2024-03-25T19:40:47.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.323+0000 7f98653f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:47.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.325+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9858001b70 con 0x7f98600a96b0 2024-03-25T19:40:47.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.325+0000 7f985dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:47.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.325+0000 7f985dffb640 1 -- 172.21.15.169:0/511619079 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9858001f80 con 0x7f98600a96b0 2024-03-25T19:40:47.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.325+0000 7f985dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.325+0000 7f985dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:47.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.325+0000 7f985dffb640 10 monclient: dump: 2024-03-25T19:40:47.326 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.325+0000 7f98653f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.325+0000 7f98653f6640 4 set_mon_vals no callback set 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.326+0000 7f98653f6640 10 monclient: shutdown 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.326+0000 7f98653f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.326+0000 7f98653f6640 1 -- 172.21.15.169:0/511619079 >> v1:172.21.15.69:6789/0 conn(0x7f98600a96b0 legacy=0x7f986013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.326+0000 7f98653f6640 1 -- 172.21.15.169:0/511619079 shutdown_connections 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.326+0000 7f98653f6640 1 -- 172.21.15.169:0/511619079 >> 172.21.15.169:0/511619079 conn(0x7f98600668e0 msgr2=0x7f9860066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.326+0000 7f98653f6640 1 -- 172.21.15.169:0/511619079 shutdown_connections 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.326+0000 7f98653f6640 1 -- 172.21.15.169:0/511619079 wait complete. 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 10 monclient: build_initial_monmap 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 10 monclient: monmap: 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:47.328 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 1 librados: starting msgr at 2024-03-25T19:40:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 1 librados: starting objecter 2024-03-25T19:40:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 1 Processor -- start 2024-03-25T19:40:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 1 -- start start 2024-03-25T19:40:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 1 librados: setting wanted keys 2024-03-25T19:40:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 1 librados: calling monclient init 2024-03-25T19:40:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.327+0000 7f98653f6640 10 monclient: init 2024-03-25T19:40:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 10 monclient(hunting): picked mon.noname-a con 0x7f98600a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 10 monclient(hunting): picked mon.noname-c con 0x7f98600b4700 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 10 monclient(hunting): picked mon.noname-b con 0x7f9860142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98601db820 con 0x7f98600a96b0 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986013cd70 con 0x7f9860142a20 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98600b1af0 con 0x7f98600b4700 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98653f6640 10 monclient(hunting): authenticate will time out at 4853.628806s 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f985effd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f98600a96b0 0x7f98601dab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.169:42802/0 (socket says 172.21.15.169:42802) 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f985effd640 1 -- 172.21.15.169:0/442493237 learned_addr learned my addr 172.21.15.169:0/442493237 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98477fe640 10 client.?.objecter ms_handle_connect 0x7f98600a96b0 2024-03-25T19:40:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.328+0000 7f98477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.329+0000 7f98477fe640 10 client.?.objecter ms_handle_connect 0x7f9860142a20 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.329+0000 7f98477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.329+0000 7f98477fe640 10 client.?.objecter ms_handle_connect 0x7f98600b4700 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.329+0000 7f98477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346124201 0 0) 0x7f98600b1af0 con 0x7f98600b4700 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f984c004eb0 con 0x7f98600b4700 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253095880 0 0) 0x7f98601db820 con 0x7f98600a96b0 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:40:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.330+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98600b1af0 con 0x7f98600a96b0 2024-03-25T19:40:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.331+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057553293 0 0) 0x7f986013cd70 con 0x7f9860142a20 2024-03-25T19:40:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.331+0000 7f98477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.331+0000 7f98477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.331+0000 7f98477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.331+0000 7f98477fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:40:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.331+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98601db820 con 0x7f9860142a20 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.335+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2680073823 0 0) 0x7f98600b1af0 con 0x7f98600a96b0 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.335+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f986013cd70 con 0x7f98600a96b0 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.335+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9854001a50 con 0x7f98600a96b0 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.335+0000 7f98477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.335+0000 7f98477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.335+0000 7f98477fe640 10 monclient(hunting): dump: 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.336 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.336 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.336 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.336 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.335+0000 7f98477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:47.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.337+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 806586953 0 0) 0x7f98601db820 con 0x7f9860142a20 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.337+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98600b1af0 con 0x7f9860142a20 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.337+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9858001e80 con 0x7f9860142a20 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.337+0000 7f98477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.337+0000 7f98477fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.337+0000 7f98477fe640 10 monclient(hunting): dump: 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.337 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.338 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2057087005 0 0) 0x7f986013cd70 con 0x7f98600a96b0 2024-03-25T19:40:47.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 >> v1:172.21.15.69:6790/0 conn(0x7f98600b4700 legacy=0x7f98601d42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 >> v1:172.21.15.169:6789/0 conn(0x7f9860142a20 legacy=0x7f98601d49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:47.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 1 monclient: found mon.a 2024-03-25T19:40:47.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:47.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98600b1c80 con 0x7f98600a96b0 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98653f6640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98653f6640 10 monclient: _renew_subs 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98653f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.338+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98601db270 con 0x7f98600a96b0 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 10 monclient: _renew_subs 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98601e3090 con 0x7f98600a96b0 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 1 librados: init done 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:47.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.339+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986004f730 con 0x7f98600a96b0 2024-03-25T19:40:47.341 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:47.341+0000 7f1af0443040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:47.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9854001bb0 con 0x7f98600a96b0 2024-03-25T19:40:47.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:47.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9854003a60 con 0x7f98600a96b0 2024-03-25T19:40:47.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:47.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 10 monclient: dump: 2024-03-25T19:40:47.342 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9854004010 con 0x7f98600a96b0 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2398586652 0 0) 0x7f9854006580 con 0x7f98600a96b0 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f985d7fa640 4 set_mon_vals no callback set 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 10 client.4125.objecter ms_dispatch 0x7f9860000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:40:47.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:40:47.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:40:47.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.342+0000 7f98477fe640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:40:47.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.437+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f98540042c0 con 0x7f98600a96b0 2024-03-25T19:40:47.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.437+0000 7f98477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.437+0000 7f98477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:47.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.510+0000 7f98653f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "68679d85-c21b-44b4-9f57-a4af14c984d1", "id": 1}] 2024-03-25T19:40:47.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.510+0000 7f98653f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "68679d85-c21b-44b4-9f57-a4af14c984d1", "id": 1}] 2024-03-25T19:40:47.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.510+0000 7f98653f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:47.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.510+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd new", "uuid": "68679d85-c21b-44b4-9f57-a4af14c984d1", "id": 1} v 0) v1 -- 0x7f9860069f30 con 0x7f98600a96b0 2024-03-25T19:40:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.550+0000 7f98477fe640 1 -- 172.21.15.169:0/442493237 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "68679d85-c21b-44b4-9f57-a4af14c984d1", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2388822697 0 940483276) 0x7f98540057c0 con 0x7f98600a96b0 2024-03-25T19:40:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.550+0000 7f98477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "68679d85-c21b-44b4-9f57-a4af14c984d1", "id": 1}] 2024-03-25T19:40:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.550+0000 7f98477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:47.550 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.550+0000 7f98653f6640 10 librados: watch_flush enter 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.550+0000 7f98653f6640 10 librados: watch_flush exit 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.550+0000 7f98653f6640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.550+0000 7f98653f6640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.551+0000 7f98653f6640 10 monclient: shutdown 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.551+0000 7f98653f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.551+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 >> v1:172.21.15.69:6789/0 conn(0x7f98600a96b0 legacy=0x7f98601dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.551+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 shutdown_connections 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.551+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 >> 172.21.15.169:0/442493237 conn(0x7f98600668e0 msgr2=0x7f98600b5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.551+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 shutdown_connections 2024-03-25T19:40:47.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.551+0000 7f98653f6640 1 -- 172.21.15.169:0/442493237 wait complete. 2024-03-25T19:40:47.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.551+0000 7f98653f6640 1 librados: shutdown 2024-03-25T19:40:47.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new a86395a9-8a87-4f9f-8589-2a4ec6c5a991 2 2024-03-25T19:40:47.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.663+0000 7fe4ceae4640 10 monclient: get_monmap_and_config 2024-03-25T19:40:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.663+0000 7fe4ceae4640 10 monclient: build_initial_monmap 2024-03-25T19:40:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.663+0000 7fe4ceae4640 10 monclient: monmap: 2024-03-25T19:40:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:40:47.664207+0000 2024-03-25T19:40:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:40:47.664207+0000 2024-03-25T19:40:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.665 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:47.665 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:47.665 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:47.665 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.663+0000 7fe4ceae4640 1 Processor -- start 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 1 -- start start 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 10 monclient: init 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c80b4700 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c8142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c80a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c80a9d60 con 0x7fe4c80a96b0 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c813ff10 con 0x7fe4c8142a20 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c80aa1f0 con 0x7fe4c80b4700 2024-03-25T19:40:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.664+0000 7fe4ceae4640 10 monclient(hunting): authenticate will time out at 4853.965285s 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.665+0000 7fe4cd081640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe4c80a96b0 0x7fe4c813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.169:42810/0 (socket says 172.21.15.169:42810) 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.665+0000 7fe4cd081640 1 -- 172.21.15.169:0/2075319578 learned_addr learned my addr 172.21.15.169:0/2075319578 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810382016 0 0) 0x7fe4c80a9d60 con 0x7fe4c80a96b0 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8003520 con 0x7fe4c80a96b0 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180065617 0 0) 0x7fe4c813ff10 con 0x7fe4c8142a20 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.667+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c80a9d60 con 0x7fe4c8142a20 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.668+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960053997 0 0) 0x7fe4c80aa1f0 con 0x7fe4c80b4700 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.668+0000 7fe4c77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.668+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.668+0000 7fe4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.668+0000 7fe4c77fe640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:40:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.668+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c813ff10 con 0x7fe4c80b4700 2024-03-25T19:40:47.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 879580481 0 0) 0x7fe4b8003520 con 0x7fe4c80a96b0 2024-03-25T19:40:47.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 >> v1:172.21.15.69:6790/0 conn(0x7fe4c80b4700 legacy=0x7fe4c8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 >> v1:172.21.15.169:6789/0 conn(0x7fe4c8142a20 legacy=0x7fe4c813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 1 monclient: found mon.noname-a 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c80b4ae0 con 0x7fe4c80a96b0 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4ceae4640 5 monclient: authenticate success, global_id 4128 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe4c0001a10 con 0x7fe4c80a96b0 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4ceae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4c77fe640 10 monclient: dump: 2024-03-25T19:40:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.670+0000 7fe4ceae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:47.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.672+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4c0001b70 con 0x7fe4c80a96b0 2024-03-25T19:40:47.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.672+0000 7fe4c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:47.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.672+0000 7fe4c77fe640 1 -- 172.21.15.169:0/2075319578 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe4c0001f80 con 0x7fe4c80a96b0 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.672+0000 7fe4ceae4640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.672+0000 7fe4ceae4640 4 set_mon_vals no callback set 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.672+0000 7fe4c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.672+0000 7fe4c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.672+0000 7fe4c77fe640 10 monclient: dump: 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.677 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.673+0000 7fe4ceae4640 10 monclient: shutdown 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.673+0000 7fe4ceae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.673+0000 7fe4ceae4640 1 -- 172.21.15.169:0/2075319578 >> v1:172.21.15.69:6789/0 conn(0x7fe4c80a96b0 legacy=0x7fe4c813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.673+0000 7fe4ceae4640 1 -- 172.21.15.169:0/2075319578 shutdown_connections 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.673+0000 7fe4ceae4640 1 -- 172.21.15.169:0/2075319578 >> 172.21.15.169:0/2075319578 conn(0x7fe4c80668e0 msgr2=0x7fe4c8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.673+0000 7fe4ceae4640 1 -- 172.21.15.169:0/2075319578 shutdown_connections 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.673+0000 7fe4ceae4640 1 -- 172.21.15.169:0/2075319578 wait complete. 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 10 monclient: build_initial_monmap 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 10 monclient: monmap: 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T19:40:47.678 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 1 librados: starting msgr at 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 1 librados: starting objecter 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 1 Processor -- start 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 1 -- start start 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 1 librados: setting wanted keys 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 1 librados: calling monclient init 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.674+0000 7fe4ceae4640 10 monclient: init 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:47.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c80a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c80b4700 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c8142a20 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c80ac830 con 0x7fe4c8142a20 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c813cc10 con 0x7fe4c80a96b0 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c813b7f0 con 0x7fe4c80b4700 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4ceae4640 10 monclient(hunting): authenticate will time out at 4853.975710s 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4cc880640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe4c80a96b0 0x7fe4c813c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:46974/0 (socket says 172.21.15.169:46974) 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4cc880640 1 -- 172.21.15.169:0/1892233210 learned_addr learned my addr 172.21.15.169:0/1892233210 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4c57fa640 10 client.?.objecter ms_handle_connect 0x7fe4c80a96b0 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4c57fa640 10 client.?.objecter ms_handle_connect 0x7fe4c8142a20 2024-03-25T19:40:47.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.675+0000 7fe4c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.676+0000 7fe4c57fa640 10 client.?.objecter ms_handle_connect 0x7fe4c80b4700 2024-03-25T19:40:47.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.676+0000 7fe4c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.677+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881547985 0 0) 0x7fe4c813cc10 con 0x7fe4c80a96b0 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.677+0000 7fe4c57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.677+0000 7fe4c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.677+0000 7fe4c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.677+0000 7fe4c57fa640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.677+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ac004eb0 con 0x7fe4c80a96b0 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.678+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459081508 0 0) 0x7fe4c813b7f0 con 0x7fe4c80b4700 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.678+0000 7fe4c57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.678+0000 7fe4c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.678+0000 7fe4c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.678+0000 7fe4c57fa640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.678+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c813cc10 con 0x7fe4c80b4700 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.682+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3034416075 0 0) 0x7fe4ac004eb0 con 0x7fe4c80a96b0 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.682+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4c813b7f0 con 0x7fe4c80a96b0 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.682+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe4bc001a10 con 0x7fe4c80a96b0 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.682+0000 7fe4c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.682+0000 7fe4c57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.682+0000 7fe4c57fa640 10 monclient(hunting): dump: 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.683 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.684 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.684 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.682+0000 7fe4c57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:47.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.683+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1902335513 0 0) 0x7fe4c813cc10 con 0x7fe4c80b4700 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.683+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4ac004eb0 con 0x7fe4c80b4700 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.683+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe4b4001910 con 0x7fe4c80b4700 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.683+0000 7fe4c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.683+0000 7fe4c57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.683+0000 7fe4c57fa640 10 monclient(hunting): dump: 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.686 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1031719472 0 0) 0x7fe4c813b7f0 con 0x7fe4c80a96b0 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 >> v1:172.21.15.69:6790/0 conn(0x7fe4c80b4700 legacy=0x7fe4c80acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 >> v1:172.21.15.69:6789/0 conn(0x7fe4c8142a20 legacy=0x7fe4c81e23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 1 monclient: found mon.b 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c813b980 con 0x7fe4c80a96b0 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:47.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 10 monclient: _renew_subs 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c80ad360 con 0x7fe4c80a96b0 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 10 monclient: _renew_subs 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c813a520 con 0x7fe4c80a96b0 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.685+0000 7fe4ceae4640 1 librados: init done 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.686+0000 7fe4ceae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.686+0000 7fe4ceae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.686+0000 7fe4ceae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.686+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c80ac630 con 0x7fe4c80a96b0 2024-03-25T19:40:47.711 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:47.712+0000 7f63ebc71040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:40:47.751 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:47.752+0000 7f1af0443040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:40:47.845 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:47.845+0000 7f63ebc71040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:40:47.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4bc001c70 con 0x7fe4c80a96b0 2024-03-25T19:40:47.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:47.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe4bc003a60 con 0x7fe4c80a96b0 2024-03-25T19:40:47.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:47.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 10 monclient: dump: 2024-03-25T19:40:47.890 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:47.890 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe4bc004010 con 0x7fe4c80a96b0 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 680172170 0 0) 0x7fe4bc0066d0 con 0x7fe4c80a96b0 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c6ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 10 client.4129.objecter ms_dispatch 0x7fe4c8000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:40:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T19:40:47.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe4bc006af0 con 0x7fe4c80a96b0 2024-03-25T19:40:47.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.890+0000 7fe4c57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:47.892 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:47.891+0000 7f1af0443040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:40:47.972 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:47.972+0000 7f63ebc71040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:40:47.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.984+0000 7fe4ceae4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a86395a9-8a87-4f9f-8589-2a4ec6c5a991", "id": 2}] 2024-03-25T19:40:47.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.984+0000 7fe4ceae4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a86395a9-8a87-4f9f-8589-2a4ec6c5a991", "id": 2}] 2024-03-25T19:40:47.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.984+0000 7fe4ceae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:47.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:47.984+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a86395a9-8a87-4f9f-8589-2a4ec6c5a991", "id": 2} v 0) v1 -- 0x7fe4c804f730 con 0x7fe4c80a96b0 2024-03-25T19:40:48.017 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:48.018+0000 7f63ebc71040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:40:48.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:48.027+0000 7f1af0443040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:40:48.073 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:48.073+0000 7f1af0443040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:40:48.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.104+0000 7fe4c57fa640 1 -- 172.21.15.169:0/1892233210 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a86395a9-8a87-4f9f-8589-2a4ec6c5a991", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3989625313 0 216599125) 0x7fe4bc0042c0 con 0x7fe4c80a96b0 2024-03-25T19:40:48.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.104+0000 7fe4c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a86395a9-8a87-4f9f-8589-2a4ec6c5a991", "id": 2}] 2024-03-25T19:40:48.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.104+0000 7fe4c57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:48.105 INFO:teuthology.orchestra.run.smithi169.stdout:2 2024-03-25T19:40:48.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 10 librados: watch_flush enter 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 10 librados: watch_flush exit 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 10 client.4129.objecter successfully canceled tick 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 10 monclient: shutdown 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 >> v1:172.21.15.169:6789/0 conn(0x7fe4c80a96b0 legacy=0x7fe4c813c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 shutdown_connections 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 >> 172.21.15.169:0/1892233210 conn(0x7fe4c80668e0 msgr2=0x7fe4c80b6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 shutdown_connections 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 1 -- 172.21.15.169:0/1892233210 wait complete. 2024-03-25T19:40:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.105+0000 7fe4ceae4640 1 librados: shutdown 2024-03-25T19:40:48.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new dae35ad2-6d51-40b6-b4fc-8d377aee1f23 3 2024-03-25T19:40:48.147 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:48.147+0000 7f63ebc71040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:48.192 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:48.193+0000 7f1af0443040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.215+0000 7fb198354640 10 monclient: get_monmap_and_config 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.215+0000 7fb198354640 10 monclient: build_initial_monmap 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.215+0000 7fb198354640 10 monclient: monmap: 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:40:48.216493+0000 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:40:48.216493+0000 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:48.216 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 1 Processor -- start 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 1 -- start start 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 10 monclient: init 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 10 monclient(hunting): picked mon.noname-c con 0x7fb1900b4700 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 10 monclient(hunting): picked mon.noname-b con 0x7fb190142a20 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 10 monclient(hunting): picked mon.noname-a con 0x7fb1900a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900a9d60 con 0x7fb1900a96b0 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19013ff10 con 0x7fb1900b4700 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900aa1f0 con 0x7fb190142a20 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:48.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.216+0000 7fb198354640 10 monclient(hunting): authenticate will time out at 4854.517481s 2024-03-25T19:40:48.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.217+0000 7fb1968f1640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb1900a96b0 0x7fb19013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.169:42828/0 (socket says 172.21.15.169:42828) 2024-03-25T19:40:48.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.217+0000 7fb1968f1640 1 -- 172.21.15.169:0/3633038609 learned_addr learned my addr 172.21.15.169:0/3633038609 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:48.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.219+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132246840 0 0) 0x7fb1900a9d60 con 0x7fb1900a96b0 2024-03-25T19:40:48.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.219+0000 7fb1950ee640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.219+0000 7fb1950ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.219+0000 7fb1950ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.219+0000 7fb1950ee640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.219+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb17c003520 con 0x7fb1900a96b0 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816316567 0 0) 0x7fb19013ff10 con 0x7fb1900b4700 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:40:48.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1900a9d60 con 0x7fb1900b4700 2024-03-25T19:40:48.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76855327 0 0) 0x7fb1900aa1f0 con 0x7fb190142a20 2024-03-25T19:40:48.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:40:48.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.220+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19013ff10 con 0x7fb190142a20 2024-03-25T19:40:48.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 130991336 0 0) 0x7fb1900a9d60 con 0x7fb1900b4700 2024-03-25T19:40:48.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 >> v1:172.21.15.69:6790/0 conn(0x7fb190142a20 legacy=0x7fb19013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 >> v1:172.21.15.69:6789/0 conn(0x7fb1900a96b0 legacy=0x7fb19013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 10 monclient: _finish_hunting 0 2024-03-25T19:40:48.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 1 monclient: found mon.noname-c 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1900b4ae0 con 0x7fb1900b4700 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 10 monclient: _finish_auth 0 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 10 monclient: _check_auth_tickets 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb18c001670 con 0x7fb1900b4700 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb1950ee640 10 monclient: dump: 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.224 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb198354640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:40:48.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.223+0000 7fb198354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:48.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb18c0017d0 con 0x7fb1900b4700 2024-03-25T19:40:48.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb1950ee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:48.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb1950ee640 1 -- 172.21.15.169:0/3633038609 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb18c001be0 con 0x7fb1900b4700 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb198354640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb198354640 4 set_mon_vals no callback set 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb1950ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb1950ee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb1950ee640 10 monclient: dump: 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.229 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb198354640 10 monclient: shutdown 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb198354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.225+0000 7fb198354640 1 -- 172.21.15.169:0/3633038609 >> v1:172.21.15.169:6789/0 conn(0x7fb1900b4700 legacy=0x7fb190142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.226+0000 7fb198354640 1 -- 172.21.15.169:0/3633038609 shutdown_connections 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.226+0000 7fb198354640 1 -- 172.21.15.169:0/3633038609 >> 172.21.15.169:0/3633038609 conn(0x7fb1900668e0 msgr2=0x7fb190066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.226+0000 7fb198354640 1 -- 172.21.15.169:0/3633038609 shutdown_connections 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.226+0000 7fb198354640 1 -- 172.21.15.169:0/3633038609 wait complete. 2024-03-25T19:40:48.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.226+0000 7fb198354640 10 monclient: build_initial_monmap 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.226+0000 7fb198354640 10 monclient: monmap: 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.226+0000 7fb198354640 1 librados: starting msgr at 2024-03-25T19:40:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.226+0000 7fb198354640 1 librados: starting objecter 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 1 Processor -- start 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 1 -- start start 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 1 librados: setting wanted keys 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 1 librados: calling monclient init 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 10 monclient: init 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 10 monclient(hunting): picked mon.noname-a con 0x7fb1900a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 10 monclient(hunting): picked mon.noname-c con 0x7fb1900b4700 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 10 monclient(hunting): picked mon.noname-b con 0x7fb190142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19012d9a0 con 0x7fb1900a96b0 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb190137700 con 0x7fb190142a20 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1901379e0 con 0x7fb1900b4700 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:48.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.227+0000 7fb198354640 10 monclient(hunting): authenticate will time out at 4854.528256s 2024-03-25T19:40:48.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.228+0000 7fb1968f1640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb190142a20 0x7fb19012d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:46996/0 (socket says 172.21.15.169:46996) 2024-03-25T19:40:48.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.228+0000 7fb1968f1640 1 -- 172.21.15.169:0/3459070439 learned_addr learned my addr 172.21.15.169:0/3459070439 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:48.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.228+0000 7fb186ffd640 10 client.?.objecter ms_handle_connect 0x7fb190142a20 2024-03-25T19:40:48.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.228+0000 7fb186ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.228+0000 7fb186ffd640 10 client.?.objecter ms_handle_connect 0x7fb1900b4700 2024-03-25T19:40:48.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.228+0000 7fb186ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.228+0000 7fb186ffd640 10 client.?.objecter ms_handle_connect 0x7fb1900a96b0 2024-03-25T19:40:48.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.228+0000 7fb186ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180059488 0 0) 0x7fb190137700 con 0x7fb190142a20 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb174004eb0 con 0x7fb190142a20 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745825806 0 0) 0x7fb1901379e0 con 0x7fb1900b4700 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.230+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190137700 con 0x7fb1900b4700 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.231+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108069279 0 0) 0x7fb19012d9a0 con 0x7fb1900a96b0 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.231+0000 7fb186ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.231+0000 7fb186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.231+0000 7fb186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.231+0000 7fb186ffd640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.231+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1901379e0 con 0x7fb1900a96b0 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.235+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1966343375 0 0) 0x7fb174004eb0 con 0x7fb190142a20 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.235+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb19012d9a0 con 0x7fb190142a20 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.235+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb188001b30 con 0x7fb190142a20 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.235+0000 7fb186ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.235+0000 7fb186ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:48.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.235+0000 7fb186ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.235+0000 7fb186ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:48.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.236+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3807219135 0 0) 0x7fb190137700 con 0x7fb1900b4700 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.236+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb174004eb0 con 0x7fb1900b4700 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.236+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb180001870 con 0x7fb1900b4700 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.236+0000 7fb186ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.236+0000 7fb186ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.236+0000 7fb186ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.240 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 746073019 0 0) 0x7fb1901379e0 con 0x7fb1900a96b0 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb190137700 con 0x7fb1900a96b0 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb18c001ae0 con 0x7fb1900a96b0 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.241 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1033567457 0 0) 0x7fb19012d9a0 con 0x7fb190142a20 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 >> v1:172.21.15.69:6790/0 conn(0x7fb1900b4700 legacy=0x7fb19012cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 >> v1:172.21.15.69:6789/0 conn(0x7fb1900a96b0 legacy=0x7fb190133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 1 monclient: found mon.b 2024-03-25T19:40:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190136550 con 0x7fb190142a20 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 10 monclient: _renew_subs 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb190133780 con 0x7fb190142a20 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 10 monclient: _renew_subs 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb190135cb0 con 0x7fb190142a20 2024-03-25T19:40:48.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.238+0000 7fb198354640 1 librados: init done 2024-03-25T19:40:48.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.239+0000 7fb198354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.239+0000 7fb198354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.239+0000 7fb198354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:48.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.239+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19004f730 con 0x7fb190142a20 2024-03-25T19:40:48.247 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:48.247+0000 7f63ebc71040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:40:48.293 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:48.294+0000 7f1af0443040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb188001fe0 con 0x7fb190142a20 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb188004d00 con 0x7fb190142a20 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 10 monclient: dump: 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb1948ed640 4 set_mon_vals no callback set 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb1880051f0 con 0x7fb190142a20 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 141222870 0 0) 0x7fb188006860 con 0x7fb190142a20 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 10 client.4135.objecter ms_dispatch 0x7fb190000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:40:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-25T19:40:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb188018de0 con 0x7fb190142a20 2024-03-25T19:40:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.344+0000 7fb186ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:48.404 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:48.404+0000 7f63ebc71040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:48.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.415+0000 7fb198354640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dae35ad2-6d51-40b6-b4fc-8d377aee1f23", "id": 3}] 2024-03-25T19:40:48.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.415+0000 7fb198354640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dae35ad2-6d51-40b6-b4fc-8d377aee1f23", "id": 3}] 2024-03-25T19:40:48.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.415+0000 7fb198354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:40:48.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.415+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dae35ad2-6d51-40b6-b4fc-8d377aee1f23", "id": 3} v 0) v1 -- 0x7fb1900b7610 con 0x7fb190142a20 2024-03-25T19:40:48.449 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:48.449+0000 7f63ebc71040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:48.452 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:48.452+0000 7f1af0443040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.477+0000 7fb186ffd640 1 -- 172.21.15.169:0/3459070439 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dae35ad2-6d51-40b6-b4fc-8d377aee1f23", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2359736667 0 525046306) 0x7fb188006480 con 0x7fb190142a20 2024-03-25T19:40:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.477+0000 7fb186ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dae35ad2-6d51-40b6-b4fc-8d377aee1f23", "id": 3}] 2024-03-25T19:40:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.477+0000 7fb186ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:48.477 INFO:teuthology.orchestra.run.smithi169.stdout:3 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 10 librados: watch_flush enter 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 10 librados: watch_flush exit 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 10 client.4135.objecter successfully canceled tick 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 10 monclient: shutdown 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 >> v1:172.21.15.169:6789/0 conn(0x7fb190142a20 legacy=0x7fb19012d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 shutdown_connections 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 >> 172.21.15.169:0/3459070439 conn(0x7fb1900668e0 msgr2=0x7fb1900b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 shutdown_connections 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 1 -- 172.21.15.169:0/3459070439 wait complete. 2024-03-25T19:40:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.478+0000 7fb198354640 1 librados: shutdown 2024-03-25T19:40:48.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 07990f28-82fb-49d8-b33d-02afbd20c66a 4 2024-03-25T19:40:48.498 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:48.499+0000 7f1af0443040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.590+0000 7f1ae079d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.590+0000 7f1ae079d640 10 monclient: build_initial_monmap 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.590+0000 7f1ae079d640 10 monclient: monmap: 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:40:48.591127+0000 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:40:48.591127+0000 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:48.590 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:48.591 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.590+0000 7f1ae079d640 1 Processor -- start 2024-03-25T19:40:48.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.590+0000 7f1ae079d640 1 -- start start 2024-03-25T19:40:48.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.590+0000 7f1ae079d640 10 monclient: init 2024-03-25T19:40:48.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:48.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:48.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 10 monclient(hunting): picked mon.noname-b con 0x7f1ad80b4700 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:48.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ad8142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:48.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ad80a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:48.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80a9d60 con 0x7f1ad80a96b0 2024-03-25T19:40:48.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad813ff10 con 0x7f1ad8142a20 2024-03-25T19:40:48.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80aa1f0 con 0x7f1ad80b4700 2024-03-25T19:40:48.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:48.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.591+0000 7f1ae079d640 10 monclient(hunting): authenticate will time out at 4854.892101s 2024-03-25T19:40:48.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.592+0000 7f1addd38640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f1ad8142a20 0x7f1ad813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:47012/0 (socket says 172.21.15.169:47012) 2024-03-25T19:40:48.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.592+0000 7f1addd38640 1 -- 172.21.15.169:0/3273093684 learned_addr learned my addr 172.21.15.169:0/3273093684 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:48.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681791485 0 0) 0x7f1ad813ff10 con 0x7f1ad8142a20 2024-03-25T19:40:48.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:40:48.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac4003520 con 0x7f1ad8142a20 2024-03-25T19:40:48.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728544259 0 0) 0x7f1ad80aa1f0 con 0x7f1ad80b4700 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad813ff10 con 0x7f1ad80b4700 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090469734 0 0) 0x7f1ad80a9d60 con 0x7f1ad80a96b0 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:40:48.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.594+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad80aa1f0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2863546198 0 0) 0x7f1ac4003520 con 0x7f1ad8142a20 2024-03-25T19:40:48.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 >> v1:172.21.15.69:6790/0 conn(0x7f1ad80b4700 legacy=0x7f1ad8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 >> v1:172.21.15.69:6789/0 conn(0x7f1ad80a96b0 legacy=0x7f1ad813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 10 monclient: _finish_hunting 0 2024-03-25T19:40:48.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 1 monclient: found mon.noname-c 2024-03-25T19:40:48.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:48.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:40:48.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad80b4ae0 con 0x7f1ad8142a20 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 10 monclient: _finish_auth 0 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 10 monclient: _check_auth_tickets 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1ac8001690 con 0x7f1ad8142a20 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1add537640 10 monclient: dump: 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.598 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1ae079d640 5 monclient: authenticate success, global_id 4138 2024-03-25T19:40:48.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.597+0000 7f1ae079d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.599+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ac80017f0 con 0x7f1ad8142a20 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.599+0000 7f1add537640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.599+0000 7f1add537640 1 -- 172.21.15.169:0/3273093684 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1ac8001c00 con 0x7f1ad8142a20 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.599+0000 7f1add537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.599+0000 7f1add537640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.599+0000 7f1add537640 10 monclient: dump: 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.602 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.599+0000 7f1ae079d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.599+0000 7f1ae079d640 4 set_mon_vals no callback set 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 10 monclient: shutdown 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 1 -- 172.21.15.169:0/3273093684 >> v1:172.21.15.169:6789/0 conn(0x7f1ad8142a20 legacy=0x7f1ad813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 1 -- 172.21.15.169:0/3273093684 shutdown_connections 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 1 -- 172.21.15.169:0/3273093684 >> 172.21.15.169:0/3273093684 conn(0x7f1ad80668e0 msgr2=0x7f1ad8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 1 -- 172.21.15.169:0/3273093684 shutdown_connections 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 1 -- 172.21.15.169:0/3273093684 wait complete. 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 10 monclient: build_initial_monmap 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.600+0000 7f1ae079d640 10 monclient: monmap: 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:48.603 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 librados: starting msgr at 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 librados: starting objecter 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 Processor -- start 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 -- start start 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 librados: setting wanted keys 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 librados: calling monclient init 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 10 monclient: init 2024-03-25T19:40:48.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ad80a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 10 monclient(hunting): picked mon.noname-b con 0x7f1ad80b4700 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ad8142a20 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80ade40 con 0x7f1ad80a96b0 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80ae0b0 con 0x7f1ad80b4700 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80ae390 con 0x7f1ad8142a20 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.601+0000 7f1ae079d640 10 monclient(hunting): authenticate will time out at 4854.902473s 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.602+0000 7f1addd38640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f1ad80b4700 0x7f1ad80ae7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:47026/0 (socket says 172.21.15.169:47026) 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.602+0000 7f1addd38640 1 -- 172.21.15.169:0/776722790 learned_addr learned my addr 172.21.15.169:0/776722790 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.602+0000 7f1aceffd640 10 client.?.objecter ms_handle_connect 0x7f1ad80b4700 2024-03-25T19:40:48.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.602+0000 7f1aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.603+0000 7f1aceffd640 10 client.?.objecter ms_handle_connect 0x7f1ad8142a20 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.603+0000 7f1aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.603+0000 7f1aceffd640 10 client.?.objecter ms_handle_connect 0x7f1ad80a96b0 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.603+0000 7f1aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531145838 0 0) 0x7f1ad80ade40 con 0x7f1ad80a96b0 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1abc004eb0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428413186 0 0) 0x7f1ad80ae390 con 0x7f1ad8142a20 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.605+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad80ade40 con 0x7f1ad8142a20 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.606+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56318083 0 0) 0x7f1ad80ae0b0 con 0x7f1ad80b4700 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.606+0000 7f1aceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.606+0000 7f1aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.606+0000 7f1aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.606+0000 7f1aceffd640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:40:48.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.606+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad80ae390 con 0x7f1ad80b4700 2024-03-25T19:40:48.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.611+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 648664325 0 0) 0x7f1abc004eb0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.611+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ad80ae0b0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.611+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1ad4001970 con 0x7f1ad80a96b0 2024-03-25T19:40:48.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.611+0000 7f1aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.611+0000 7f1aceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:48.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.611+0000 7f1aceffd640 10 monclient(hunting): dump: 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.611+0000 7f1aceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:48.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.612+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1440619592 0 0) 0x7f1ad80ade40 con 0x7f1ad8142a20 2024-03-25T19:40:48.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.612+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1abc004eb0 con 0x7f1ad8142a20 2024-03-25T19:40:48.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.612+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1ad0001b50 con 0x7f1ad8142a20 2024-03-25T19:40:48.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.612+0000 7f1aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.612+0000 7f1aceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.612+0000 7f1aceffd640 10 monclient(hunting): dump: 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.614+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 556405287 0 0) 0x7f1ad80ae390 con 0x7f1ad80b4700 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.614+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ad80ade40 con 0x7f1ad80b4700 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.614+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1ac8001b00 con 0x7f1ad80b4700 2024-03-25T19:40:48.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.614+0000 7f1aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.614+0000 7f1aceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.614+0000 7f1aceffd640 10 monclient(hunting): dump: 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.615 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.616 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:48.614+0000 7f63ebc71040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3307355354 0 0) 0x7f1ad80ae0b0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 >> v1:172.21.15.69:6790/0 conn(0x7f1ad8142a20 legacy=0x7f1ad81e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 >> v1:172.21.15.169:6789/0 conn(0x7f1ad80b4700 legacy=0x7f1ad80ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 1 monclient: found mon.a 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad80ad060 con 0x7f1ad80a96b0 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1aceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 5 monclient: authenticate success, global_id 4143 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 10 monclient: _renew_subs 2024-03-25T19:40:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad80abcc0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 10 monclient: _renew_subs 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad80ac960 con 0x7f1ad80a96b0 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.615+0000 7f1ae079d640 1 librados: init done 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.616+0000 7f1ae079d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.616+0000 7f1ae079d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.616+0000 7f1ae079d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.616+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad804f730 con 0x7f1ad80a96b0 2024-03-25T19:40:48.666 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:48.667+0000 7f1af0443040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.710+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ad40035f0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.710+0000 7f1aceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.710+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1ad4003900 con 0x7f1ad80a96b0 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.710+0000 7f1aceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.710+0000 7f1aceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.710+0000 7f1aceffd640 10 monclient: dump: 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.710 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.711 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.711 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.711 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.711 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.711 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.710+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1ad4003e30 con 0x7f1ad80a96b0 2024-03-25T19:40:48.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.710+0000 7f1adcd36640 4 set_mon_vals no callback set 2024-03-25T19:40:48.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.711+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2093401831 0 0) 0x7f1ad40055c0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.711+0000 7f1aceffd640 10 client.4143.objecter ms_dispatch 0x7f1ad8000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:40:48.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.711+0000 7f1aceffd640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:40:48.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.711+0000 7f1aceffd640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:40:48.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.711+0000 7f1aceffd640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T19:40:48.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.711+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1ad40050b0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.711+0000 7f1aceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.711+0000 7f1aceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:48.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.783+0000 7f1ae079d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "07990f28-82fb-49d8-b33d-02afbd20c66a", "id": 4}] 2024-03-25T19:40:48.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.783+0000 7f1ae079d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "07990f28-82fb-49d8-b33d-02afbd20c66a", "id": 4}] 2024-03-25T19:40:48.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.783+0000 7f1ae079d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:48.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.783+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd new", "uuid": "07990f28-82fb-49d8-b33d-02afbd20c66a", "id": 4} v 0) v1 -- 0x7f1ad813e5d0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.834+0000 7f1aceffd640 1 -- 172.21.15.169:0/776722790 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07990f28-82fb-49d8-b33d-02afbd20c66a", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4259993168 0 1697012583) 0x7f1ad40053a0 con 0x7f1ad80a96b0 2024-03-25T19:40:48.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.834+0000 7f1aceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "07990f28-82fb-49d8-b33d-02afbd20c66a", "id": 4}] 2024-03-25T19:40:48.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.834+0000 7f1aceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:48.834 INFO:teuthology.orchestra.run.smithi169.stdout:4 2024-03-25T19:40:48.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 10 librados: watch_flush enter 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 10 librados: watch_flush exit 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 10 client.4143.objecter successfully canceled tick 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 10 monclient: shutdown 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 >> v1:172.21.15.69:6789/0 conn(0x7f1ad80a96b0 legacy=0x7f1ad813aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 shutdown_connections 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 >> 172.21.15.169:0/776722790 conn(0x7f1ad80668e0 msgr2=0x7f1ad80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 shutdown_connections 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 1 -- 172.21.15.169:0/776722790 wait complete. 2024-03-25T19:40:48.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.835+0000 7f1ae079d640 1 librados: shutdown 2024-03-25T19:40:48.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new df15ac59-21c6-41e8-8f03-eeb135940c6b 5 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.987+0000 7f8d290ae640 10 monclient: get_monmap_and_config 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.987+0000 7f8d290ae640 10 monclient: build_initial_monmap 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.988+0000 7f8d290ae640 10 monclient: monmap: 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:40:48.988919+0000 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:40:48.988919+0000 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:48.988 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:48.989 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.988+0000 7f8d290ae640 1 Processor -- start 2024-03-25T19:40:48.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.988+0000 7f8d290ae640 1 -- start start 2024-03-25T19:40:48.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.988+0000 7f8d290ae640 10 monclient: init 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 10 monclient(hunting): picked mon.noname-a con 0x7f8d240b4700 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 10 monclient(hunting): picked mon.noname-c con 0x7f8d24142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 10 monclient(hunting): picked mon.noname-b con 0x7f8d240a96b0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240a9d60 con 0x7f8d240b4700 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2413ff10 con 0x7f8d24142a20 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240aa1f0 con 0x7f8d240a96b0 2024-03-25T19:40:48.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:48.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d290ae640 10 monclient(hunting): authenticate will time out at 4855.289811s 2024-03-25T19:40:48.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d2359e640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f8d240a96b0 0x7f8d2413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.169:37718/0 (socket says 172.21.15.169:37718) 2024-03-25T19:40:48.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.989+0000 7f8d2359e640 1 -- 172.21.15.169:0/136204966 learned_addr learned my addr 172.21.15.169:0/136204966 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:48.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716072553 0 0) 0x7f8d240aa1f0 con 0x7f8d240a96b0 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d10003520 con 0x7f8d240a96b0 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880295576 0 0) 0x7f8d240a9d60 con 0x7f8d240b4700 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:40:48.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d240aa1f0 con 0x7f8d240b4700 2024-03-25T19:40:48.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930732254 0 0) 0x7f8d2413ff10 con 0x7f8d24142a20 2024-03-25T19:40:48.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:40:48.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.992+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d240a9d60 con 0x7f8d24142a20 2024-03-25T19:40:48.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804388392 0 0) 0x7f8d10003520 con 0x7f8d240a96b0 2024-03-25T19:40:48.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 >> v1:172.21.15.169:6789/0 conn(0x7f8d24142a20 legacy=0x7f8d2413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 >> v1:172.21.15.69:6789/0 conn(0x7f8d240b4700 legacy=0x7f8d24142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 1 monclient: found mon.noname-b 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240b4ae0 con 0x7f8d240a96b0 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 10 monclient: _finish_auth 0 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8d14001690 con 0x7f8d240a96b0 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d21d9b640 10 monclient: dump: 2024-03-25T19:40:48.996 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d290ae640 5 monclient: authenticate success, global_id 4142 2024-03-25T19:40:48.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:48.995+0000 7f8d290ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:49.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.022+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d140017f0 con 0x7f8d240a96b0 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.022+0000 7f8d21d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.022+0000 7f8d21d9b640 1 -- 172.21.15.169:0/136204966 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8d14001c00 con 0x7f8d240a96b0 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.022+0000 7f8d21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.022+0000 7f8d21d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.022+0000 7f8d21d9b640 10 monclient: dump: 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.022 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.023 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.022+0000 7f8d290ae640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:49.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.022+0000 7f8d290ae640 4 set_mon_vals no callback set 2024-03-25T19:40:49.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.023+0000 7f8d290ae640 10 monclient: shutdown 2024-03-25T19:40:49.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.023+0000 7f8d290ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:49.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.023+0000 7f8d290ae640 1 -- 172.21.15.169:0/136204966 >> v1:172.21.15.69:6790/0 conn(0x7f8d240a96b0 legacy=0x7f8d2413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.023+0000 7f8d290ae640 1 -- 172.21.15.169:0/136204966 shutdown_connections 2024-03-25T19:40:49.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.023+0000 7f8d290ae640 1 -- 172.21.15.169:0/136204966 >> 172.21.15.169:0/136204966 conn(0x7f8d240668e0 msgr2=0x7f8d24066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:49.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.023+0000 7f8d290ae640 1 -- 172.21.15.169:0/136204966 shutdown_connections 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.023+0000 7f8d290ae640 1 -- 172.21.15.169:0/136204966 wait complete. 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.023+0000 7f8d290ae640 10 monclient: build_initial_monmap 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.024+0000 7f8d290ae640 10 monclient: monmap: 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.024+0000 7f8d290ae640 1 librados: starting msgr at 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.024+0000 7f8d290ae640 1 librados: starting objecter 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.024+0000 7f8d290ae640 1 Processor -- start 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.024+0000 7f8d290ae640 1 -- start start 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.024+0000 7f8d290ae640 1 librados: setting wanted keys 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.024+0000 7f8d290ae640 1 librados: calling monclient init 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.024+0000 7f8d290ae640 10 monclient: init 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 10 monclient(hunting): picked mon.noname-b con 0x7f8d240a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 10 monclient(hunting): picked mon.noname-a con 0x7f8d240b4700 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 10 monclient(hunting): picked mon.noname-c con 0x7f8d24142a20 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d241db820 con 0x7f8d240b4700 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2413cd70 con 0x7f8d240a96b0 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240b1af0 con 0x7f8d24142a20 2024-03-25T19:40:49.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d290ae640 10 monclient(hunting): authenticate will time out at 4855.325756s 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d22d9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f8d240a96b0 0x7f8d241dab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:47032/0 (socket says 172.21.15.169:47032) 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.025+0000 7f8d22d9d640 1 -- 172.21.15.169:0/3535419278 learned_addr learned my addr 172.21.15.169:0/3535419278 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.026+0000 7f8d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f8d240a96b0 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.026+0000 7f8d1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.026+0000 7f8d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f8d24142a20 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.026+0000 7f8d1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.026+0000 7f8d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f8d240b4700 2024-03-25T19:40:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.026+0000 7f8d1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448904914 0 0) 0x7f8d240b1af0 con 0x7f8d24142a20 2024-03-25T19:40:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:40:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d04004eb0 con 0x7f8d24142a20 2024-03-25T19:40:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881555717 0 0) 0x7f8d2413cd70 con 0x7f8d240a96b0 2024-03-25T19:40:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d240b1af0 con 0x7f8d240a96b0 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.028+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885879378 0 0) 0x7f8d241db820 con 0x7f8d240b4700 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.029+0000 7f8d1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.029+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.029+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.029+0000 7f8d1b7fe640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:40:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.029+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d2413cd70 con 0x7f8d240b4700 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.033+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281046186 0 0) 0x7f8d04004eb0 con 0x7f8d24142a20 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.033+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d241db820 con 0x7f8d24142a20 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.033+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8d14001b00 con 0x7f8d24142a20 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.033+0000 7f8d1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.033+0000 7f8d1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.033+0000 7f8d1b7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.034 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.034 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.034 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.033+0000 7f8d1b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.034+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 493564620 0 0) 0x7f8d2413cd70 con 0x7f8d240b4700 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.034+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d04004eb0 con 0x7f8d240b4700 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.034+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8d0c001830 con 0x7f8d240b4700 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.034+0000 7f8d1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.034+0000 7f8d1b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.034+0000 7f8d1b7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.035+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3193552524 0 0) 0x7f8d240b1af0 con 0x7f8d240a96b0 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.035+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d2413cd70 con 0x7f8d240a96b0 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.035+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8d1c001d30 con 0x7f8d240a96b0 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.035+0000 7f8d1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.035+0000 7f8d1b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.035+0000 7f8d1b7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.036 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.037 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.037 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.037 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.037 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.037 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.037 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2957067601 0 0) 0x7f8d04004eb0 con 0x7f8d240b4700 2024-03-25T19:40:49.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 >> v1:172.21.15.69:6790/0 conn(0x7f8d24142a20 legacy=0x7f8d241d49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 >> v1:172.21.15.169:6789/0 conn(0x7f8d240a96b0 legacy=0x7f8d241dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:49.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 1 monclient: found mon.a 2024-03-25T19:40:49.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240b1c80 con 0x7f8d240b4700 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 5 monclient: authenticate success, global_id 4149 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 10 monclient: _renew_subs 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d241db270 con 0x7f8d240b4700 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 10 monclient: _renew_subs 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d241e3090 con 0x7f8d240b4700 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.037+0000 7f8d290ae640 1 librados: init done 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.038+0000 7f8d290ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.038+0000 7f8d290ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.038+0000 7f8d290ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.038+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2404f730 con 0x7f8d240b4700 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d0c001d40 con 0x7f8d240b4700 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8d0c0037b0 con 0x7f8d240b4700 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 10 monclient: dump: 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d2159a640 4 set_mon_vals no callback set 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8d0c003d60 con 0x7f8d240b4700 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 170205995 0 0) 0x7f8d0c0054a0 con 0x7f8d240b4700 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 10 client.4149.objecter ms_dispatch 0x7f8d24000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:40:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.043+0000 7f8d1b7fe640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T19:40:49.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.136+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8d0c005030 con 0x7f8d240b4700 2024-03-25T19:40:49.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.136+0000 7f8d1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.136+0000 7f8d1b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:49.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.208+0000 7f8d290ae640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "df15ac59-21c6-41e8-8f03-eeb135940c6b", "id": 5}] 2024-03-25T19:40:49.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.208+0000 7f8d290ae640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "df15ac59-21c6-41e8-8f03-eeb135940c6b", "id": 5}] 2024-03-25T19:40:49.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.208+0000 7f8d290ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.208+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd new", "uuid": "df15ac59-21c6-41e8-8f03-eeb135940c6b", "id": 5} v 0) v1 -- 0x7f8d24069f30 con 0x7f8d240b4700 2024-03-25T19:40:49.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.238+0000 7f8d1b7fe640 1 -- 172.21.15.169:0/3535419278 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "df15ac59-21c6-41e8-8f03-eeb135940c6b", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3775990258 0 1988412176) 0x7f8d0c0052a0 con 0x7f8d240b4700 2024-03-25T19:40:49.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.238+0000 7f8d1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "df15ac59-21c6-41e8-8f03-eeb135940c6b", "id": 5}] 2024-03-25T19:40:49.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.238+0000 7f8d1b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:49.238 INFO:teuthology.orchestra.run.smithi169.stdout:5 2024-03-25T19:40:49.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 10 librados: watch_flush enter 2024-03-25T19:40:49.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 10 librados: watch_flush exit 2024-03-25T19:40:49.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T19:40:49.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 10 client.4149.objecter successfully canceled tick 2024-03-25T19:40:49.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 10 monclient: shutdown 2024-03-25T19:40:49.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:49.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 >> v1:172.21.15.69:6789/0 conn(0x7f8d240b4700 legacy=0x7f8d241d42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 shutdown_connections 2024-03-25T19:40:49.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 >> 172.21.15.169:0/3535419278 conn(0x7f8d240668e0 msgr2=0x7f8d240b5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:49.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 shutdown_connections 2024-03-25T19:40:49.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 1 -- 172.21.15.169:0/3535419278 wait complete. 2024-03-25T19:40:49.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.239+0000 7f8d290ae640 1 librados: shutdown 2024-03-25T19:40:49.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new c7d6304e-042b-4de5-a93d-983af03ad23a 6 2024-03-25T19:40:49.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.352+0000 7f46041cf640 10 monclient: get_monmap_and_config 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.352+0000 7f46041cf640 10 monclient: build_initial_monmap 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.352+0000 7f46041cf640 10 monclient: monmap: 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:40:49.353343+0000 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:40:49.353343+0000 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:49.352 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.352+0000 7f46041cf640 1 Processor -- start 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 1 -- start start 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 10 monclient: init 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 10 monclient(hunting): picked mon.noname-b con 0x7f45fc0b4700 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 10 monclient(hunting): picked mon.noname-a con 0x7f45fc142a20 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 10 monclient(hunting): picked mon.noname-c con 0x7f45fc0a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:49.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc0a9d60 con 0x7f45fc142a20 2024-03-25T19:40:49.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc13ff10 con 0x7f45fc0a96b0 2024-03-25T19:40:49.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc0aa1f0 con 0x7f45fc0b4700 2024-03-25T19:40:49.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:49.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.353+0000 7f46041cf640 10 monclient(hunting): authenticate will time out at 4855.654411s 2024-03-25T19:40:49.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.354+0000 7f460276c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f45fc0a96b0 0x7f45fc13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:47048/0 (socket says 172.21.15.169:47048) 2024-03-25T19:40:49.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.354+0000 7f460276c640 1 -- 172.21.15.169:0/3244909918 learned_addr learned my addr 172.21.15.169:0/3244909918 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748016589 0 0) 0x7f45fc0a9d60 con 0x7f45fc142a20 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e8003520 con 0x7f45fc142a20 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484817684 0 0) 0x7f45fc0aa1f0 con 0x7f45fc0b4700 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45fc0a9d60 con 0x7f45fc0b4700 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951039660 0 0) 0x7f45fc13ff10 con 0x7f45fc0a96b0 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:40:49.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.357+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45fc0aa1f0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3276377882 0 0) 0x7f45fc0a9d60 con 0x7f45fc0b4700 2024-03-25T19:40:49.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 >> v1:172.21.15.169:6789/0 conn(0x7f45fc0a96b0 legacy=0x7f45fc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 >> v1:172.21.15.69:6789/0 conn(0x7f45fc142a20 legacy=0x7f45fc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 10 monclient: _finish_hunting 0 2024-03-25T19:40:49.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 1 monclient: found mon.noname-b 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc0b4ae0 con 0x7f45fc0b4700 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 10 monclient: _finish_auth 0 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 10 monclient: _check_auth_tickets 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f46041cf640 5 monclient: authenticate success, global_id 4148 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f46041cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f45f8001630 con 0x7f45fc0b4700 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f4600f69640 10 monclient: dump: 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.361 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.362 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.362 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.362 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.362 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.362 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.362 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.360+0000 7f46041cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:49.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45f80018c0 con 0x7f45fc0b4700 2024-03-25T19:40:49.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f4600f69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:49.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f4600f69640 1 -- 172.21.15.169:0/3244909918 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f45f8001c50 con 0x7f45fc0b4700 2024-03-25T19:40:49.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f4600f69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f4600f69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f4600f69640 10 monclient: dump: 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f46041cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f46041cf640 4 set_mon_vals no callback set 2024-03-25T19:40:49.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f46041cf640 10 monclient: shutdown 2024-03-25T19:40:49.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f46041cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:49.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f46041cf640 1 -- 172.21.15.169:0/3244909918 >> v1:172.21.15.69:6790/0 conn(0x7f45fc0b4700 legacy=0x7f45fc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f46041cf640 1 -- 172.21.15.169:0/3244909918 shutdown_connections 2024-03-25T19:40:49.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.363+0000 7f46041cf640 1 -- 172.21.15.169:0/3244909918 >> 172.21.15.169:0/3244909918 conn(0x7f45fc0668e0 msgr2=0x7f45fc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:49.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.364+0000 7f46041cf640 1 -- 172.21.15.169:0/3244909918 shutdown_connections 2024-03-25T19:40:49.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.364+0000 7f46041cf640 1 -- 172.21.15.169:0/3244909918 wait complete. 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.364+0000 7f46041cf640 10 monclient: build_initial_monmap 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.364+0000 7f46041cf640 10 monclient: monmap: 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T19:40:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.364+0000 7f46041cf640 1 librados: starting msgr at 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.364+0000 7f46041cf640 1 librados: starting objecter 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 1 Processor -- start 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 1 -- start start 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 1 librados: setting wanted keys 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 1 librados: calling monclient init 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 10 monclient: init 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 10 monclient(hunting): picked mon.noname-a con 0x7f45fc0a96b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 10 monclient(hunting): picked mon.noname-b con 0x7f45fc0b4700 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 10 monclient(hunting): picked mon.noname-c con 0x7f45fc142a20 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc139490 con 0x7f45fc0a96b0 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc137be0 con 0x7f45fc0b4700 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc13a6b0 con 0x7f45fc142a20 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.365+0000 7f46041cf640 10 monclient(hunting): authenticate will time out at 4855.666223s 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.366+0000 7f4601f6b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f45fc0a96b0 0x7f45fc1da900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.169:42914/0 (socket says 172.21.15.169:42914) 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.366+0000 7f4601f6b640 1 -- 172.21.15.169:0/84624333 learned_addr learned my addr 172.21.15.169:0/84624333 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.366+0000 7f45f27fc640 10 client.?.objecter ms_handle_connect 0x7f45fc0a96b0 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.366+0000 7f45f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.366+0000 7f45f27fc640 10 client.?.objecter ms_handle_connect 0x7f45fc0b4700 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.366+0000 7f45f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.366+0000 7f45f27fc640 10 client.?.objecter ms_handle_connect 0x7f45fc142a20 2024-03-25T19:40:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.366+0000 7f45f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171955805 0 0) 0x7f45fc139490 con 0x7f45fc0a96b0 2024-03-25T19:40:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e0004eb0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957478299 0 0) 0x7f45fc13a6b0 con 0x7f45fc142a20 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.368+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45fc139490 con 0x7f45fc142a20 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.369+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342164797 0 0) 0x7f45fc137be0 con 0x7f45fc0b4700 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.369+0000 7f45f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.369+0000 7f45f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.369+0000 7f45f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.369+0000 7f45f27fc640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:40:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.369+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45fc13a6b0 con 0x7f45fc0b4700 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 840292574 0 0) 0x7f45e0004eb0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45fc137be0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f45f8001630 con 0x7f45fc0a96b0 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 10 monclient(hunting): dump: 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.373 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1221002792 0 0) 0x7f45fc139490 con 0x7f45fc142a20 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45e0004eb0 con 0x7f45fc142a20 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f45f4001b50 con 0x7f45fc142a20 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.373+0000 7f45f27fc640 10 monclient(hunting): dump: 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.374 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.375 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.375 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.375 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.375 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.375 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.375 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.375 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.375+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1258145309 0 0) 0x7f45fc13a6b0 con 0x7f45fc0b4700 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.375+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45fc139490 con 0x7f45fc0b4700 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.375+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f45ec001230 con 0x7f45fc0b4700 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.375+0000 7f45f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.375+0000 7f45f27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.375+0000 7f45f27fc640 10 monclient(hunting): dump: 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.377 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4201074903 0 0) 0x7f45fc137be0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 >> v1:172.21.15.69:6790/0 conn(0x7f45fc142a20 legacy=0x7f45fc1374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 >> v1:172.21.15.169:6789/0 conn(0x7f45fc0b4700 legacy=0x7f45fc136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 1 monclient: found mon.a 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc138f70 con 0x7f45fc0a96b0 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:49.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f45f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 5 monclient: authenticate success, global_id 4155 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 10 monclient: _renew_subs 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc138a90 con 0x7f45fc0a96b0 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 10 monclient: _renew_subs 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc1386f0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.376+0000 7f46041cf640 1 librados: init done 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.377+0000 7f46041cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.377+0000 7f46041cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.377+0000 7f46041cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.377+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc138d90 con 0x7f45fc0a96b0 2024-03-25T19:40:49.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45f8001250 con 0x7f45fc0a96b0 2024-03-25T19:40:49.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:49.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f45f8003b40 con 0x7f45fc0a96b0 2024-03-25T19:40:49.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 10 monclient: dump: 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f45f80040f0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f3fff640 4 set_mon_vals no callback set 2024-03-25T19:40:49.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3679865971 0 0) 0x7f45f80069c0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 10 client.4155.objecter ms_dispatch 0x7f45fc000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:40:49.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:40:49.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:40:49.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.380+0000 7f45f27fc640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T19:40:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.476+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f45f80043a0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.476+0000 7f45f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.476+0000 7f45f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:49.532 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:49.533+0000 7f1af0443040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:40:49.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.548+0000 7f46041cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c7d6304e-042b-4de5-a93d-983af03ad23a", "id": 6}] 2024-03-25T19:40:49.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.548+0000 7f46041cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c7d6304e-042b-4de5-a93d-983af03ad23a", "id": 6}] 2024-03-25T19:40:49.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.548+0000 7f46041cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:49.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.548+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c7d6304e-042b-4de5-a93d-983af03ad23a", "id": 6} v 0) v1 -- 0x7f45fc04f730 con 0x7f45fc0a96b0 2024-03-25T19:40:49.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.576+0000 7f45f27fc640 1 -- 172.21.15.169:0/84624333 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c7d6304e-042b-4de5-a93d-983af03ad23a", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3696273913 0 1113811849) 0x7f45f8006bd0 con 0x7f45fc0a96b0 2024-03-25T19:40:49.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.576+0000 7f45f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c7d6304e-042b-4de5-a93d-983af03ad23a", "id": 6}] 2024-03-25T19:40:49.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.576+0000 7f45f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:49.576 INFO:teuthology.orchestra.run.smithi169.stdout:6 2024-03-25T19:40:49.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 10 librados: watch_flush enter 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 10 librados: watch_flush exit 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 10 client.4155.objecter successfully canceled tick 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 10 monclient: shutdown 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 >> v1:172.21.15.69:6789/0 conn(0x7f45fc0a96b0 legacy=0x7f45fc1da900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 shutdown_connections 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 >> 172.21.15.169:0/84624333 conn(0x7f45fc0668e0 msgr2=0x7f45fc13f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 shutdown_connections 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 1 -- 172.21.15.169:0/84624333 wait complete. 2024-03-25T19:40:49.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.577+0000 7f46041cf640 1 librados: shutdown 2024-03-25T19:40:49.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 16becd40-0b67-4e42-b407-63f5b24cd430 7 2024-03-25T19:40:49.680 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:49.680+0000 7f63ebc71040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:40:49.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.687+0000 7fdaf08f6640 10 monclient: get_monmap_and_config 2024-03-25T19:40:49.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.687+0000 7fdaf08f6640 10 monclient: build_initial_monmap 2024-03-25T19:40:49.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.687+0000 7fdaf08f6640 10 monclient: monmap: 2024-03-25T19:40:49.687 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:49.687 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.687 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:40:49.688466+0000 2024-03-25T19:40:49.687 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:40:49.688466+0000 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 1 Processor -- start 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 1 -- start start 2024-03-25T19:40:49.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 10 monclient: init 2024-03-25T19:40:49.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:49.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdaec0b4700 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdaec142a20 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdaec0a96b0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0a9d60 con 0x7fdaec142a20 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec13ff10 con 0x7fdaec0b4700 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0aa1f0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.688+0000 7fdaf08f6640 10 monclient(hunting): authenticate will time out at 4855.989444s 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.689+0000 7fdaea59c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fdaec0b4700 0x7fdaec142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:47076/0 (socket says 172.21.15.169:47076) 2024-03-25T19:40:49.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.689+0000 7fdaea59c640 1 -- 172.21.15.169:0/4294094236 learned_addr learned my addr 172.21.15.169:0/4294094236 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:49.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.691+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203043270 0 0) 0x7fdaec0aa1f0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:40:49.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad4003520 con 0x7fdaec0a96b0 2024-03-25T19:40:49.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401333759 0 0) 0x7fdaec0a9d60 con 0x7fdaec142a20 2024-03-25T19:40:49.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:40:49.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaec0aa1f0 con 0x7fdaec142a20 2024-03-25T19:40:49.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197415897 0 0) 0x7fdaec13ff10 con 0x7fdaec0b4700 2024-03-25T19:40:49.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:40:49.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.692+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaec0a9d60 con 0x7fdaec0b4700 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2565065351 0 0) 0x7fdad4003520 con 0x7fdaec0a96b0 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 >> v1:172.21.15.169:6789/0 conn(0x7fdaec0b4700 legacy=0x7fdaec142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 >> v1:172.21.15.69:6789/0 conn(0x7fdaec142a20 legacy=0x7fdaec13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 10 monclient: _finish_hunting 0 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 1 monclient: found mon.noname-b 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec0b4ae0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 10 monclient: _finish_auth 0 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 10 monclient: _check_auth_tickets 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdae4001a10 con 0x7fdaec0a96b0 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdaf08f6640 5 monclient: authenticate success, global_id 4154 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdaf08f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdae959a640 10 monclient: dump: 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.696 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.697 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.697 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.697 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.697 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.697 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.695+0000 7fdaf08f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:49.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdae4001b70 con 0x7fdaec0a96b0 2024-03-25T19:40:49.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdae959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:49.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdae959a640 1 -- 172.21.15.169:0/4294094236 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdae4001f80 con 0x7fdaec0a96b0 2024-03-25T19:40:49.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:49.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 4 set_mon_vals no callback set 2024-03-25T19:40:49.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdae959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdae959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:49.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdae959a640 10 monclient: dump: 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 10 monclient: shutdown 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 1 -- 172.21.15.169:0/4294094236 >> v1:172.21.15.69:6790/0 conn(0x7fdaec0a96b0 legacy=0x7fdaec13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 1 -- 172.21.15.169:0/4294094236 shutdown_connections 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 1 -- 172.21.15.169:0/4294094236 >> 172.21.15.169:0/4294094236 conn(0x7fdaec0668e0 msgr2=0x7fdaec066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:49.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 1 -- 172.21.15.169:0/4294094236 shutdown_connections 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.697+0000 7fdaf08f6640 1 -- 172.21.15.169:0/4294094236 wait complete. 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.698+0000 7fdaf08f6640 10 monclient: build_initial_monmap 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.698+0000 7fdaf08f6640 10 monclient: monmap: 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.698+0000 7fdaf08f6640 1 librados: starting msgr at 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.698+0000 7fdaf08f6640 1 librados: starting objecter 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 1 Processor -- start 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 1 -- start start 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 1 librados: setting wanted keys 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 1 librados: calling monclient init 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 10 monclient: init 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdaec0a96b0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdaec0b4700 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdaec142a20 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:49.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec13afc0 con 0x7fdaec0b4700 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec13ccf0 con 0x7fdaec142a20 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec13b2a0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.699+0000 7fdaf08f6640 10 monclient(hunting): authenticate will time out at 4856.000246s 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.700+0000 7fdae9d9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdaec0b4700 0x7fdaec133f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.169:42920/0 (socket says 172.21.15.169:42920) 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.700+0000 7fdae9d9b640 1 -- 172.21.15.169:0/3816000300 learned_addr learned my addr 172.21.15.169:0/3816000300 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.700+0000 7fdadaffd640 10 client.?.objecter ms_handle_connect 0x7fdaec0b4700 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.700+0000 7fdadaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.700+0000 7fdadaffd640 10 client.?.objecter ms_handle_connect 0x7fdaec142a20 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.700+0000 7fdadaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.700+0000 7fdadaffd640 10 client.?.objecter ms_handle_connect 0x7fdaec0a96b0 2024-03-25T19:40:49.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.700+0000 7fdadaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:49.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.702+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352639683 0 0) 0x7fdaec13b2a0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.702+0000 7fdadaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.702+0000 7fdadaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.702+0000 7fdadaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.702+0000 7fdadaffd640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.702+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc004eb0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583013734 0 0) 0x7fdaec13afc0 con 0x7fdaec0b4700 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaec13b2a0 con 0x7fdaec0b4700 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111295042 0 0) 0x7fdaec13ccf0 con 0x7fdaec142a20 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:40:49.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.703+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaec13afc0 con 0x7fdaec142a20 2024-03-25T19:40:49.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.707+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184185928 0 0) 0x7fdacc004eb0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.707+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaec13ccf0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.707+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdadc0018f0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.707+0000 7fdadaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.707+0000 7fdadaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:49.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.707+0000 7fdadaffd640 10 monclient(hunting): dump: 2024-03-25T19:40:49.707 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.708 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.708 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.708 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.708 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.708 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.708 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.709 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.709 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.709 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.707+0000 7fdadaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:49.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.709+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2032721802 0 0) 0x7fdaec13b2a0 con 0x7fdaec0b4700 2024-03-25T19:40:49.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.709+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdacc004eb0 con 0x7fdaec0b4700 2024-03-25T19:40:49.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.709+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdae00019b0 con 0x7fdaec0b4700 2024-03-25T19:40:49.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.709+0000 7fdadaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.709+0000 7fdadaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:49.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.709+0000 7fdadaffd640 10 monclient(hunting): dump: 2024-03-25T19:40:49.711 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396230510 0 0) 0x7fdaec13afc0 con 0x7fdaec142a20 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaec13b2a0 con 0x7fdaec142a20 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdae4001e80 con 0x7fdaec142a20 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:49.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 10 monclient(hunting): dump: 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 914195316 0 0) 0x7fdaec13ccf0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 >> v1:172.21.15.169:6789/0 conn(0x7fdaec142a20 legacy=0x7fdaec1d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 >> v1:172.21.15.69:6789/0 conn(0x7fdaec0b4700 legacy=0x7fdaec133f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:49.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 1 monclient: found mon.c 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec13c3e0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdadaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 5 monclient: authenticate success, global_id 4157 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 10 monclient: _renew_subs 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec14e6c0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 10 monclient: _renew_subs 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:49.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec134830 con 0x7fdaec0a96b0 2024-03-25T19:40:49.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.710+0000 7fdaf08f6640 1 librados: init done 2024-03-25T19:40:49.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.711+0000 7fdaf08f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.711+0000 7fdaf08f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.711+0000 7fdaf08f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:49.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.711+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec04f730 con 0x7fdaec0a96b0 2024-03-25T19:40:49.833 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:49.833+0000 7f1af0443040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:40:49.883 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:49.884+0000 7f1af0443040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:40:49.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdadc001d40 con 0x7fdaec0a96b0 2024-03-25T19:40:49.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:49.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdadc004a00 con 0x7fdaec0a96b0 2024-03-25T19:40:49.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 10 monclient: dump: 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:40:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdadc004fb0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2761073018 0 0) 0x7fdadc0067d0 con 0x7fdaec0a96b0 2024-03-25T19:40:49.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdae8d99640 4 set_mon_vals no callback set 2024-03-25T19:40:49.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 10 client.4157.objecter ms_dispatch 0x7fdaec000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:40:49.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 3 client.4157.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:40:49.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.898+0000 7fdadaffd640 3 client.4157.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:40:49.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.899+0000 7fdadaffd640 20 client.4157.objecter dump_active .. 0 homeless 2024-03-25T19:40:49.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.901+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdadc006280 con 0x7fdaec0a96b0 2024-03-25T19:40:49.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.901+0000 7fdadaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:49.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.901+0000 7fdadaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:49.973 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:49.973+0000 7f63ebc71040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:40:49.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.974+0000 7fdaf08f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "16becd40-0b67-4e42-b407-63f5b24cd430", "id": 7}] 2024-03-25T19:40:49.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.974+0000 7fdaf08f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "16becd40-0b67-4e42-b407-63f5b24cd430", "id": 7}] 2024-03-25T19:40:49.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.974+0000 7fdaf08f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:49.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:49.974+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd new", "uuid": "16becd40-0b67-4e42-b407-63f5b24cd430", "id": 7} v 0) v1 -- 0x7fdaec069f30 con 0x7fdaec0a96b0 2024-03-25T19:40:50.024 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:50.025+0000 7f1af0443040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:40:50.028 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:50.028+0000 7f63ebc71040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:40:50.151 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-03-25T19:40:50.151+0000 7f63ebc71040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:40:50.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdadaffd640 1 -- 172.21.15.169:0/3816000300 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "16becd40-0b67-4e42-b407-63f5b24cd430", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 4000907369 0 1371665406) 0x7fdadc006580 con 0x7fdaec0a96b0 2024-03-25T19:40:50.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdadaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "16becd40-0b67-4e42-b407-63f5b24cd430", "id": 7}] 2024-03-25T19:40:50.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdadaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:50.162 INFO:teuthology.orchestra.run.smithi169.stdout:7 2024-03-25T19:40:50.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 10 librados: watch_flush enter 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 10 librados: watch_flush exit 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 20 client.4157.objecter shutdown clearing up homeless session... 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 10 client.4157.objecter successfully canceled tick 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 10 monclient: shutdown 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 >> v1:172.21.15.69:6790/0 conn(0x7fdaec0a96b0 legacy=0x7fdaec13bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 shutdown_connections 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 >> 172.21.15.169:0/3816000300 conn(0x7fdaec0668e0 msgr2=0x7fdaec0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 shutdown_connections 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 1 -- 172.21.15.169:0/3816000300 wait complete. 2024-03-25T19:40:50.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T19:40:50.163+0000 7fdaf08f6640 1 librados: shutdown 2024-03-25T19:40:50.173 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:50.173 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:40:50.174 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:40:50.176 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:40:50.176 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:50.176 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:40:50.176 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:40:50.178 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:40:50.179 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:50.179 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:40:50.179 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:40:50.181 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:40:50.181 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:50.181 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:40:50.181 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:40:50.183 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:40:50.184 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:50.184 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:40:50.184 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:40:50.186 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:40:50.186 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:50.187 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:40:50.187 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:40:50.189 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:40:50.189 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:50.189 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:40:50.189 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:40:50.191 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:40:50.191 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:50.191 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:40:50.191 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:40:50.193 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:40:50.194 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.340+0000 7fdbe76f1640 10 monclient: get_monmap_and_config 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.340+0000 7fdbe76f1640 10 monclient: build_initial_monmap 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.340+0000 7fdbe76f1640 10 monclient: monmap: 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:40:50.342101+0000 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:40:50.342101+0000 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:50.341 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:50.342 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:50.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.341+0000 7fdbe76f1640 1 Processor -- start 2024-03-25T19:40:50.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.342+0000 7fdbe76f1640 1 -- start start 2024-03-25T19:40:50.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.342+0000 7fdbe76f1640 10 monclient: init 2024-03-25T19:40:50.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.342+0000 7fdbe76f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:50.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.342+0000 7fdbe76f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:50.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.342+0000 7fdbe76f1640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe00a9920 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.342+0000 7fdbe76f1640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe00b45a0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.342+0000 7fdbe76f1640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe0146450 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.342+0000 7fdbe76f1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe006b100 con 0x7fdbe0146450 2024-03-25T19:40:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.343+0000 7fdbe76f1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00b2940 con 0x7fdbe00a9920 2024-03-25T19:40:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.343+0000 7fdbe76f1640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe013fec0 con 0x7fdbe00b45a0 2024-03-25T19:40:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.343+0000 7fdbe76f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.343+0000 7fdbe76f1640 10 monclient(hunting): authenticate will time out at 4862.611624s 2024-03-25T19:40:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.344+0000 7fdbe4c8c640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fdbe00b45a0 0x7fdbe0145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:51230/0 (socket says 172.21.15.69:51230) 2024-03-25T19:40:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.344+0000 7fdbe4c8c640 1 -- 172.21.15.69:0/204991170 learned_addr learned my addr 172.21.15.69:0/204991170 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:40:50.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.366+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060144206 0 0) 0x7fdbe006b100 con 0x7fdbe0146450 2024-03-25T19:40:50.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.366+0000 7fdbd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.366+0000 7fdbd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.366+0000 7fdbd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.366+0000 7fdbd7fff640 10 monclient(hunting): my global_id is 4164 2024-03-25T19:40:50.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.366+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbcc003520 con 0x7fdbe0146450 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.371+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834685187 0 0) 0x7fdbcc003520 con 0x7fdbe0146450 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.371+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 >> v1:172.21.15.69:6790/0 conn(0x7fdbe00b45a0 legacy=0x7fdbe0145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 >> v1:172.21.15.169:6789/0 conn(0x7fdbe00a9920 legacy=0x7fdbe013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 1 monclient: found mon.noname-a 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0140050 con 0x7fdbe0146450 2024-03-25T19:40:50.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdbd8001a10 con 0x7fdbe0146450 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbd7fff640 10 monclient: dump: 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:50.375 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbe76f1640 5 monclient: authenticate success, global_id 4164 2024-03-25T19:40:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.372+0000 7fdbe76f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.374+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd8001ca0 con 0x7fdbe0146450 2024-03-25T19:40:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.374+0000 7fdbd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.374+0000 7fdbe76f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.374+0000 7fdbe76f1640 4 set_mon_vals no callback set 2024-03-25T19:40:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.375+0000 7fdbd7fff640 1 -- 172.21.15.69:0/204991170 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdbd8001e90 con 0x7fdbe0146450 2024-03-25T19:40:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.375+0000 7fdbd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.375+0000 7fdbd7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.375+0000 7fdbd7fff640 10 monclient: dump: 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.375+0000 7fdbe76f1640 10 monclient: shutdown 2024-03-25T19:40:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.375+0000 7fdbe76f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.375+0000 7fdbe76f1640 1 -- 172.21.15.69:0/204991170 >> v1:172.21.15.69:6789/0 conn(0x7fdbe0146450 legacy=0x7fdbe014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.376+0000 7fdbe76f1640 1 -- 172.21.15.69:0/204991170 shutdown_connections 2024-03-25T19:40:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.376+0000 7fdbe76f1640 1 -- 172.21.15.69:0/204991170 >> 172.21.15.69:0/204991170 conn(0x7fdbe00b7f40 msgr2=0x7fdbe00b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.376+0000 7fdbe76f1640 1 -- 172.21.15.69:0/204991170 shutdown_connections 2024-03-25T19:40:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.376+0000 7fdbe76f1640 1 -- 172.21.15.69:0/204991170 wait complete. 2024-03-25T19:40:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.377+0000 7fdbe76f1640 10 monclient: build_initial_monmap 2024-03-25T19:40:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.377+0000 7fdbe76f1640 10 monclient: monmap: 2024-03-25T19:40:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-03-25T19:40:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-03-25T19:40:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-03-25T19:40:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.377+0000 7fdbe76f1640 1 librados: starting msgr at 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.377+0000 7fdbe76f1640 1 librados: starting objecter 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.378+0000 7fdbe76f1640 1 Processor -- start 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.378+0000 7fdbe76f1640 1 -- start start 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.378+0000 7fdbe76f1640 1 librados: setting wanted keys 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.378+0000 7fdbe76f1640 1 librados: calling monclient init 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.378+0000 7fdbe76f1640 10 monclient: init 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.378+0000 7fdbe76f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.378+0000 7fdbe76f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.378+0000 7fdbe76f1640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe00a9920 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.379+0000 7fdbe76f1640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe00b45a0 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.379+0000 7fdbe76f1640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe0146450 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.379+0000 7fdbe76f1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe01428b0 con 0x7fdbe0146450 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.379+0000 7fdbe76f1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe0141740 con 0x7fdbe00b45a0 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.379+0000 7fdbe76f1640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe01418d0 con 0x7fdbe00a9920 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.379+0000 7fdbe76f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.379+0000 7fdbe76f1640 10 monclient(hunting): authenticate will time out at 4862.648307s 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.380+0000 7fdbe5c8e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdbe0146450 0x7fdbe01db000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59436/0 (socket says 172.21.15.69:59436) 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.380+0000 7fdbe5c8e640 1 -- 172.21.15.69:0/2162765080 learned_addr learned my addr 172.21.15.69:0/2162765080 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.380+0000 7fdbd5ffb640 10 client.?.objecter ms_handle_connect 0x7fdbe0146450 2024-03-25T19:40:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.380+0000 7fdbd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.381+0000 7fdbd5ffb640 10 client.?.objecter ms_handle_connect 0x7fdbe00a9920 2024-03-25T19:40:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.381+0000 7fdbd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.384+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836077616 0 0) 0x7fdbe01418d0 con 0x7fdbe00a9920 2024-03-25T19:40:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.384+0000 7fdbd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.384+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.384+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.384+0000 7fdbd5ffb640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:40:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.384+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc4004eb0 con 0x7fdbe00a9920 2024-03-25T19:40:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.388+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882069481 0 0) 0x7fdbe01428b0 con 0x7fdbe0146450 2024-03-25T19:40:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.388+0000 7fdbd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.388+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.388+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.388+0000 7fdbd5ffb640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:40:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.388+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe01418d0 con 0x7fdbe0146450 2024-03-25T19:40:50.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.392+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2228109748 0 0) 0x7fdbc4004eb0 con 0x7fdbe00a9920 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.392+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe01428b0 con 0x7fdbe00a9920 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.392+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdbdc001510 con 0x7fdbe00a9920 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.392+0000 7fdbd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.392+0000 7fdbd5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.392+0000 7fdbd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:50.392 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:50.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.392+0000 7fdbd5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.395+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2303805023 0 0) 0x7fdbe01428b0 con 0x7fdbe00a9920 2024-03-25T19:40:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 >> v1:172.21.15.169:6789/0 conn(0x7fdbe00b45a0 legacy=0x7fdbe01da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 >> v1:172.21.15.69:6789/0 conn(0x7fdbe0146450 legacy=0x7fdbe01db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 1 monclient: found mon.c 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0137760 con 0x7fdbe00a9920 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 5 monclient: authenticate success, global_id 4160 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 10 monclient: _renew_subs 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe0137950 con 0x7fdbe00a9920 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 10 monclient: _renew_subs 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe013bd30 con 0x7fdbe00a9920 2024-03-25T19:40:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.396+0000 7fdbe76f1640 1 librados: init done 2024-03-25T19:40:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.398+0000 7fdbe76f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.398+0000 7fdbe76f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.398+0000 7fdbe76f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.398+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe004f730 con 0x7fdbe00a9920 2024-03-25T19:40:50.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.585+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbdc001a20 con 0x7fdbe00a9920 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.585+0000 7fdbd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.585+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fdbdc003910 con 0x7fdbe00a9920 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.585+0000 7fdbd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.585+0000 7fdbd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.585+0000 7fdbd5ffb640 10 monclient: dump: 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:50.587 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:40:50.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.585+0000 7fdbd77fe640 4 set_mon_vals no callback set 2024-03-25T19:40:50.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.586+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2003511902 0 0) 0x7fdbdc00e9f0 con 0x7fdbe00a9920 2024-03-25T19:40:50.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.590+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fdbdc014590 con 0x7fdbe00a9920 2024-03-25T19:40:50.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.590+0000 7fdbd5ffb640 10 client.4160.objecter ms_dispatch 0x7fdbe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:50.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.590+0000 7fdbd5ffb640 3 client.4160.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:50.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.590+0000 7fdbd5ffb640 3 client.4160.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:50.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.590+0000 7fdbd5ffb640 20 client.4160.objecter dump_active .. 0 homeless 2024-03-25T19:40:50.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.590+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdbdc013a30 con 0x7fdbe00a9920 2024-03-25T19:40:50.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.590+0000 7fdbd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:50.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.590+0000 7fdbd5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.714+0000 7fdbe76f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.714+0000 7fdbe76f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.714+0000 7fdbe76f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:40:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.714+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbe00b5ee0 con 0x7fdbe00a9920 2024-03-25T19:40:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.804+0000 7fdbd5ffb640 1 -- 172.21.15.69:0/2162765080 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fdbdc01c570 con 0x7fdbe00a9920 2024-03-25T19:40:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.804+0000 7fdbd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.804+0000 7fdbd5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:50.805 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-25T19:40:50.805 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.805+0000 7fdbe76f1640 10 librados: watch_flush enter 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.805+0000 7fdbe76f1640 10 librados: watch_flush exit 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.805+0000 7fdbe76f1640 20 client.4160.objecter shutdown clearing up homeless session... 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.805+0000 7fdbe76f1640 10 client.4160.objecter successfully canceled tick 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.806+0000 7fdbe76f1640 10 monclient: shutdown 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.806+0000 7fdbe76f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.806+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 >> v1:172.21.15.69:6790/0 conn(0x7fdbe00a9920 legacy=0x7fdbe0137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.806+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 shutdown_connections 2024-03-25T19:40:50.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.806+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 >> 172.21.15.69:0/2162765080 conn(0x7fdbe00b7f40 msgr2=0x7fdbe00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:50.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.806+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 shutdown_connections 2024-03-25T19:40:50.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.806+0000 7fdbe76f1640 1 -- 172.21.15.69:0/2162765080 wait complete. 2024-03-25T19:40:50.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-25T19:40:50.806+0000 7fdbe76f1640 1 librados: shutdown 2024-03-25T19:40:50.822 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:40:50.822 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:40:51.628 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.776+0000 7f1f1f29a640 10 monclient: get_monmap_and_config 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.776+0000 7f1f1f29a640 10 monclient: build_initial_monmap 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.776+0000 7f1f1f29a640 10 monclient: monmap: 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:40:51.778005+0000 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:40:51.778005+0000 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:51.777 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:51.778 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:51.778 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.777+0000 7f1f1f29a640 1 Processor -- start 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.778+0000 7f1f1f29a640 1 -- start start 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.778+0000 7f1f1f29a640 10 monclient: init 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.778+0000 7f1f1f29a640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.778+0000 7f1f1f29a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.779+0000 7f1f1f29a640 10 monclient(hunting): picked mon.noname-c con 0x7f1f180a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.779+0000 7f1f1f29a640 10 monclient(hunting): picked mon.noname-b con 0x7f1f1813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.779+0000 7f1f1f29a640 10 monclient(hunting): picked mon.noname-a con 0x7f1f181430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.779+0000 7f1f1f29a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f18142520 con 0x7f1f181430c0 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.779+0000 7f1f1f29a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f180b28a0 con 0x7f1f180a98f0 2024-03-25T19:40:51.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.779+0000 7f1f1f29a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f180b2a30 con 0x7f1f1813f710 2024-03-25T19:40:51.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.779+0000 7f1f1f29a640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:51.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.779+0000 7f1f1f29a640 10 monclient(hunting): authenticate will time out at 4864.047993s 2024-03-25T19:40:51.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.810+0000 7f1f1d036640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f1f180a98f0 0x7f1f180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:47138/0 (socket says 172.21.15.69:47138) 2024-03-25T19:40:51.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.810+0000 7f1f1d036640 1 -- 172.21.15.69:0/2888197516 learned_addr learned my addr 172.21.15.69:0/2888197516 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:40:51.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.819+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100965239 0 0) 0x7f1f18142520 con 0x7f1f181430c0 2024-03-25T19:40:51.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.819+0000 7f1f0ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.819+0000 7f1f0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.819+0000 7f1f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.819+0000 7f1f0ffff640 10 monclient(hunting): my global_id is 4182 2024-03-25T19:40:51.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.819+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f04003520 con 0x7f1f181430c0 2024-03-25T19:40:51.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 34375679 0 0) 0x7f1f04003520 con 0x7f1f181430c0 2024-03-25T19:40:51.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 >> v1:172.21.15.69:6790/0 conn(0x7f1f1813f710 legacy=0x7f1f181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 >> v1:172.21.15.169:6789/0 conn(0x7f1f180a98f0 legacy=0x7f1f180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 1 monclient: found mon.noname-a 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180b44c0 con 0x7f1f181430c0 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f1f29a640 5 monclient: authenticate success, global_id 4182 2024-03-25T19:40:51.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.826+0000 7f1f1f29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.827+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1f10001a10 con 0x7f1f181430c0 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.827+0000 7f1f0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.827+0000 7f1f0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.827+0000 7f1f0ffff640 10 monclient: dump: 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.828 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:51.829 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:51.829 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:51.829 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:51.829 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:51.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.827+0000 7f1f1f29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:51.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.830+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f10001c60 con 0x7f1f181430c0 2024-03-25T19:40:51.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.830+0000 7f1f0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:51.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.830+0000 7f1f1f29a640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:51.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.830+0000 7f1f1f29a640 4 set_mon_vals no callback set 2024-03-25T19:40:51.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f1f29a640 10 monclient: shutdown 2024-03-25T19:40:51.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f1f29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:51.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f0ffff640 1 -- 172.21.15.69:0/2888197516 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1f10001ff0 con 0x7f1f181430c0 2024-03-25T19:40:51.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f1f29a640 1 -- 172.21.15.69:0/2888197516 >> v1:172.21.15.69:6789/0 conn(0x7f1f181430c0 legacy=0x7f1f1814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f0ffff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-25T19:40:51.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f1f29a640 1 -- 172.21.15.69:0/2888197516 shutdown_connections 2024-03-25T19:40:51.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f1f29a640 1 -- 172.21.15.69:0/2888197516 >> 172.21.15.69:0/2888197516 conn(0x7f1f180b7ea0 msgr2=0x7f1f180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:51.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f1f29a640 1 -- 172.21.15.69:0/2888197516 shutdown_connections 2024-03-25T19:40:51.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.831+0000 7f1f1f29a640 1 -- 172.21.15.69:0/2888197516 wait complete. 2024-03-25T19:40:51.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.833+0000 7f1f1f29a640 10 monclient: build_initial_monmap 2024-03-25T19:40:51.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.833+0000 7f1f1f29a640 10 monclient: monmap: 2024-03-25T19:40:51.833 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:40:51.833 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:51.834 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:40:51.834 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:40:51.834 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:51.834 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:40:51.834 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:51.834 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:51.834 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:51.834 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:51.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.834+0000 7f1f1f29a640 1 librados: starting msgr at 2024-03-25T19:40:51.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.834+0000 7f1f1f29a640 1 librados: starting objecter 2024-03-25T19:40:51.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.834+0000 7f1f1f29a640 1 Processor -- start 2024-03-25T19:40:51.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.834+0000 7f1f1f29a640 1 -- start start 2024-03-25T19:40:51.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.834+0000 7f1f1f29a640 1 librados: setting wanted keys 2024-03-25T19:40:51.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.834+0000 7f1f1f29a640 1 librados: calling monclient init 2024-03-25T19:40:51.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.834+0000 7f1f1f29a640 10 monclient: init 2024-03-25T19:40:51.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.835+0000 7f1f1f29a640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:51.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.835+0000 7f1f1f29a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:51.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.836+0000 7f1f1f29a640 10 monclient(hunting): picked mon.noname-c con 0x7f1f180a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:51.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.836+0000 7f1f1f29a640 10 monclient(hunting): picked mon.noname-b con 0x7f1f1813f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:51.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.836+0000 7f1f1f29a640 10 monclient(hunting): picked mon.noname-a con 0x7f1f181430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:51.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.836+0000 7f1f1f29a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f18136810 con 0x7f1f181430c0 2024-03-25T19:40:51.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.836+0000 7f1f1f29a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f181319b0 con 0x7f1f1813f710 2024-03-25T19:40:51.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.836+0000 7f1f1f29a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f18136a80 con 0x7f1f180a98f0 2024-03-25T19:40:51.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.836+0000 7f1f1f29a640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:51.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.837+0000 7f1f1f29a640 10 monclient(hunting): authenticate will time out at 4864.105661s 2024-03-25T19:40:51.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.841+0000 7f1f1d837640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1f181430c0 0x7f1f181312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35178/0 (socket says 172.21.15.69:35178) 2024-03-25T19:40:51.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.841+0000 7f1f1d837640 1 -- 172.21.15.69:0/132285259 learned_addr learned my addr 172.21.15.69:0/132285259 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:40:51.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.843+0000 7f1f0dffb640 10 client.?.objecter ms_handle_connect 0x7f1f181430c0 2024-03-25T19:40:51.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.843+0000 7f1f0dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:51.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.850+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241422613 0 0) 0x7f1f18136810 con 0x7f1f181430c0 2024-03-25T19:40:51.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.850+0000 7f1f0dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.850+0000 7f1f0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.850+0000 7f1f0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.850+0000 7f1f0dffb640 10 monclient(hunting): my global_id is 4185 2024-03-25T19:40:51.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.850+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1efc004eb0 con 0x7f1f181430c0 2024-03-25T19:40:51.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.855+0000 7f1f0dffb640 10 client.?.objecter ms_handle_connect 0x7f1f1813f710 2024-03-25T19:40:51.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.855+0000 7f1f0dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:51.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.857+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941662978 0 0) 0x7f1efc004eb0 con 0x7f1f181430c0 2024-03-25T19:40:51.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.858+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f18136810 con 0x7f1f181430c0 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.859+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1f100020b0 con 0x7f1f181430c0 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.859+0000 7f1f0dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.859+0000 7f1f0dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.859+0000 7f1f0dffb640 10 monclient(hunting): dump: 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:51.859 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:51.860 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:51.860 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:51.860 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:51.860 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.859+0000 7f1f0dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:51.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.865+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3225200069 0 0) 0x7f1f18136810 con 0x7f1f181430c0 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 >> v1:172.21.15.69:6790/0 conn(0x7f1f180a98f0 legacy=0x7f1f181d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 >> v1:172.21.15.169:6789/0 conn(0x7f1f1813f710 legacy=0x7f1f18130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 1 monclient: found mon.a 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180b7860 con 0x7f1f181430c0 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 5 monclient: authenticate success, global_id 4185 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 10 monclient: _renew_subs 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f180b7530 con 0x7f1f181430c0 2024-03-25T19:40:51.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:51.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 10 monclient: _renew_subs 2024-03-25T19:40:51.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:51.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f18135400 con 0x7f1f181430c0 2024-03-25T19:40:51.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.866+0000 7f1f1f29a640 1 librados: init done 2024-03-25T19:40:51.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.867+0000 7f1f1f29a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:51.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.867+0000 7f1f1f29a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:51.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.867+0000 7f1f1f29a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:51.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.867+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f180627b0 con 0x7f1f181430c0 2024-03-25T19:40:51.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.869+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f10001590 con 0x7f1f181430c0 2024-03-25T19:40:51.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.869+0000 7f1f0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:51.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.869+0000 7f1f0f7fe640 4 set_mon_vals no callback set 2024-03-25T19:40:51.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.870+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1f10003ec0 con 0x7f1f181430c0 2024-03-25T19:40:51.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.870+0000 7f1f0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.871+0000 7f1f0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:51.871+0000 7f1f0dffb640 10 monclient: dump: 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:51.871 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:51.890 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:51.890+0000 7f1a911d5640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:40:51.905 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T19:40:51.905+0000 7f1a911d5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:40:52.259 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.259+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3045255052 0 0) 0x7f1f100040a0 con 0x7f1f181430c0 2024-03-25T19:40:52.259 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.260+0000 7f1f1d036640 10 monclient: get_auth_request con 0x7f1efc06da80 auth_method 0 2024-03-25T19:40:52.260 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.260+0000 7f1f0dffb640 10 client.4185.objecter ms_handle_connect 0x7f1efc06da80 2024-03-25T19:40:52.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.267+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f1f10054d40 con 0x7f1f181430c0 2024-03-25T19:40:52.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.267+0000 7f1f0dffb640 10 client.4185.objecter ms_dispatch 0x7f1f18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:52.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.267+0000 7f1f0dffb640 3 client.4185.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:52.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.267+0000 7f1f0dffb640 3 client.4185.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:52.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.267+0000 7f1f0dffb640 20 client.4185.objecter dump_active .. 0 homeless 2024-03-25T19:40:52.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.267+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f10050680 con 0x7f1f181430c0 2024-03-25T19:40:52.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.267+0000 7f1f0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:52.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.267+0000 7f1f0dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:52.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.506+0000 7f1f1f29a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:52.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.506+0000 7f1f1f29a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:52.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.506+0000 7f1f1f29a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:52.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.506+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f180b76c0 con 0x7f1f181430c0 2024-03-25T19:40:52.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.633+0000 7f1f0dffb640 1 -- 172.21.15.69:0/132285259 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f1f180b76c0 con 0x7f1f181430c0 2024-03-25T19:40:52.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.633+0000 7f1f0dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:52.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.633+0000 7f1f0dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:52.633 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:40:52.634 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 10 librados: watch_flush enter 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 10 librados: watch_flush exit 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 20 client.4185.objecter shutdown clearing up homeless session... 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 10 client.4185.objecter successfully canceled tick 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 >> v1:172.21.15.169:6800/3072548259 conn(0x7f1efc06da80 legacy=0x7f1efc06ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 10 monclient: shutdown 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 >> v1:172.21.15.69:6789/0 conn(0x7f1f181430c0 legacy=0x7f1f181312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 shutdown_connections 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.636+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 >> 172.21.15.69:0/132285259 conn(0x7f1f180b7ea0 msgr2=0x7f1f18074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.637+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 shutdown_connections 2024-03-25T19:40:52.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.637+0000 7f1f1f29a640 1 -- 172.21.15.69:0/132285259 wait complete. 2024-03-25T19:40:52.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:52.637+0000 7f1f1f29a640 1 librados: shutdown 2024-03-25T19:40:52.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:56.964 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:06.712 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.029 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:06.777 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.177 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:06.919 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.237 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:06.979 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.342 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:07.085 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.379 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:07.115 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.402 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:07.145 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.422 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:07.153 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.439 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:07.176 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.483 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:07.215 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.497 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:07.219 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.556 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:07.312 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.557 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:07.279 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.589 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-25T19:40:57.584+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:57.614 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:07.369 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.722 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:07.453 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.779 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:07.511 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.785 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-25T19:40:57.780+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:57.961 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-25T19:40:57.955+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:58.014 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-03-25T19:40:58.008+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:58.038 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-03-25T19:40:58.033+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:58.053 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:07.801 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.149 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-25T19:40:58.144+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:58.189 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-03-25T19:40:58.181+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:58.299 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:08.041 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.366 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-03-25T19:40:58.360+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:58.420 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:08.168 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.437 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:08.180 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.478 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:08.215 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.495 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.227 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.610 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:08.332 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.662 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:08.417 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.663 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:08.406 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.736 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:08.484 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.804 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:08.547 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.823 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:08.554 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.850 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:08.587 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.868 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.599 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.974 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:08.697 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.975 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.706 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:58.979 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:08.722 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.009 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:08.757 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.031 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:08.786 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.156 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:08.898 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.185 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:08.917 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.215 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:08.952 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.247 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:08.989 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.263 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.994 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.282 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:09.030 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.300 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:09.023 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.382 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.138 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.443 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:09.186 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.456 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:59.511 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:09.253 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.520 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.256 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.546 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:09.294 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.555 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.287 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.556+0000 7f13a7b9a640 10 monclient: get_monmap_and_config 2024-03-25T19:40:59.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.556+0000 7f13a7b9a640 10 monclient: build_initial_monmap 2024-03-25T19:40:59.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.556+0000 7f13a7b9a640 10 monclient: monmap: 2024-03-25T19:40:59.556 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:40:59.556 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:59.556 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:40:59.557394+0000 2024-03-25T19:40:59.556 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:40:59.557394+0000 2024-03-25T19:40:59.556 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:59.557 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:59.557 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:59.557 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:40:59.557 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:40:59.557 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:59.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.556+0000 7f13a7b9a640 1 Processor -- start 2024-03-25T19:40:59.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.556+0000 7f13a7b9a640 1 -- start start 2024-03-25T19:40:59.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.556+0000 7f13a7b9a640 10 monclient: init 2024-03-25T19:40:59.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:59.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:59.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 10 monclient(hunting): picked mon.noname-c con 0x7f13a00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 10 monclient(hunting): picked mon.noname-a con 0x7f13a00b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 10 monclient(hunting): picked mon.noname-b con 0x7f13a014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a00af940 con 0x7f13a00b1730 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a00b0a70 con 0x7f13a00b4d60 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a00b0d50 con 0x7f13a014be90 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.557+0000 7f13a7b9a640 10 monclient(hunting): authenticate will time out at 4871.826230s 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.558+0000 7f13a6137640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f13a014be90 0x7f13a014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:43760/0 (socket says 172.21.15.69:43760) 2024-03-25T19:40:59.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.558+0000 7f13a6137640 1 -- 172.21.15.69:0/2247416726 learned_addr learned my addr 172.21.15.69:0/2247416726 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905511867 0 0) 0x7f13a00b0a70 con 0x7f13a00b4d60 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 10 monclient(hunting): my global_id is 4195 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f138c003520 con 0x7f13a00b4d60 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250312079 0 0) 0x7f13a00af940 con 0x7f13a00b1730 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.560+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a00b0a70 con 0x7f13a00b1730 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.561+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653190990 0 0) 0x7f13a00b0d50 con 0x7f13a014be90 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.561+0000 7f13a4934640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.561+0000 7f13a4934640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.561+0000 7f13a4934640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.561+0000 7f13a4934640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:40:59.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.561+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a00af940 con 0x7f13a014be90 2024-03-25T19:40:59.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222264372 0 0) 0x7f138c003520 con 0x7f13a00b4d60 2024-03-25T19:40:59.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 >> v1:172.21.15.69:6790/0 conn(0x7f13a014be90 legacy=0x7f13a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 >> v1:172.21.15.69:6789/0 conn(0x7f13a00b1730 legacy=0x7f13a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 10 monclient: _finish_hunting 0 2024-03-25T19:40:59.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 1 monclient: found mon.noname-c 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a00b1030 con 0x7f13a00b4d60 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 10 monclient: _finish_auth 0 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 10 monclient: _check_auth_tickets 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f139c001690 con 0x7f13a00b4d60 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a4934640 10 monclient: dump: 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.564 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:59.565 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:59.565 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:59.565 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:59.565 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:59.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a7b9a640 5 monclient: authenticate success, global_id 4195 2024-03-25T19:40:59.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.563+0000 7f13a7b9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:59.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f139c0017f0 con 0x7f13a00b4d60 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a4934640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a7b9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a4934640 1 -- 172.21.15.69:0/2247416726 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f139c001c00 con 0x7f13a00b4d60 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a7b9a640 4 set_mon_vals no callback set 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a4934640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a4934640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a4934640 10 monclient: dump: 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:59.567 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a7b9a640 10 monclient: shutdown 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a7b9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.565+0000 7f13a7b9a640 1 -- 172.21.15.69:0/2247416726 >> v1:172.21.15.169:6789/0 conn(0x7f13a00b4d60 legacy=0x7f13a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.566+0000 7f13a7b9a640 1 -- 172.21.15.69:0/2247416726 shutdown_connections 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.566+0000 7f13a7b9a640 1 -- 172.21.15.69:0/2247416726 >> 172.21.15.69:0/2247416726 conn(0x7f13a00617a0 msgr2=0x7f13a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.566+0000 7f13a7b9a640 1 -- 172.21.15.69:0/2247416726 shutdown_connections 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.566+0000 7f13a7b9a640 1 -- 172.21.15.69:0/2247416726 wait complete. 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.566+0000 7f13a7b9a640 10 monclient: build_initial_monmap 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.566+0000 7f13a7b9a640 10 monclient: monmap: 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:40:59.568 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 librados: starting msgr at 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 librados: starting objecter 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 Processor -- start 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 -- start start 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 librados: setting wanted keys 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 librados: calling monclient init 2024-03-25T19:40:59.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 10 monclient: init 2024-03-25T19:40:59.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 10 monclient(hunting): picked mon.noname-a con 0x7f13a00b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 10 monclient(hunting): picked mon.noname-b con 0x7f13a00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 10 monclient(hunting): picked mon.noname-c con 0x7f13a014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a013acc0 con 0x7f13a00b1730 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a01d8620 con 0x7f13a00b4d60 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a013af30 con 0x7f13a014be90 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.567+0000 7f13a7b9a640 10 monclient(hunting): authenticate will time out at 4871.836410s 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.568+0000 7f13a6137640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f13a014be90 0x7f13a01e30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:43766/0 (socket says 172.21.15.69:43766) 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.568+0000 7f13a6137640 1 -- 172.21.15.69:0/3641168912 learned_addr learned my addr 172.21.15.69:0/3641168912 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.568+0000 7f13967fc640 10 client.?.objecter ms_handle_connect 0x7f13a014be90 2024-03-25T19:40:59.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.568+0000 7f13967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.568+0000 7f13967fc640 10 client.?.objecter ms_handle_connect 0x7f13a00b4d60 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.568+0000 7f13967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.568+0000 7f13967fc640 10 client.?.objecter ms_handle_connect 0x7f13a00b1730 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.568+0000 7f13967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.569+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633751585 0 0) 0x7f13a01d8620 con 0x7f13a00b4d60 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.569+0000 7f13967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.569+0000 7f13967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.569+0000 7f13967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.569+0000 7f13967fc640 10 monclient(hunting): my global_id is 4198 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.570+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1384004eb0 con 0x7f13a00b4d60 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.570+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722644611 0 0) 0x7f13a013af30 con 0x7f13a014be90 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.570+0000 7f13967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.570+0000 7f13967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.570+0000 7f13967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.570+0000 7f13967fc640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:40:59.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.570+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a01d8620 con 0x7f13a014be90 2024-03-25T19:40:59.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.571+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876358610 0 0) 0x7f13a013acc0 con 0x7f13a00b1730 2024-03-25T19:40:59.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.571+0000 7f13967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.571+0000 7f13967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.571+0000 7f13967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.571+0000 7f13967fc640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:40:59.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.571+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a013af30 con 0x7f13a00b1730 2024-03-25T19:40:59.575 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:09.295 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.576 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.304 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.575+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 516429226 0 0) 0x7f13a013af30 con 0x7f13a00b1730 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.575+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13a013acc0 con 0x7f13a00b1730 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.575+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f139c001b00 con 0x7f13a00b1730 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.575+0000 7f13967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.575+0000 7f13967fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.575+0000 7f13967fc640 10 monclient(hunting): dump: 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.577 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.575+0000 7f13967fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.576+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3086139427 0 0) 0x7f13a01d8620 con 0x7f13a014be90 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.576+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13a013af30 con 0x7f13a014be90 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.576+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1398001b90 con 0x7f13a014be90 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.576+0000 7f13967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.576+0000 7f13967fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.576+0000 7f13967fc640 10 monclient(hunting): dump: 2024-03-25T19:40:59.578 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.577+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 615899943 0 0) 0x7f1384004eb0 con 0x7f13a00b4d60 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.577+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13a01d8620 con 0x7f13a00b4d60 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.577+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1390001970 con 0x7f13a00b4d60 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.577+0000 7f13967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.577+0000 7f13967fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.577+0000 7f13967fc640 10 monclient(hunting): dump: 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:59.580 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.579+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3897848807 0 0) 0x7f13a013acc0 con 0x7f13a00b1730 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.579+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 >> v1:172.21.15.69:6790/0 conn(0x7f13a014be90 legacy=0x7f13a01e30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 >> v1:172.21.15.169:6789/0 conn(0x7f13a00b4d60 legacy=0x7f13a013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 1 monclient: found mon.a 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a01d79f0 con 0x7f13a00b1730 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:59.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 5 monclient: authenticate success, global_id 4215 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 10 monclient: _renew_subs 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a013c310 con 0x7f13a00b1730 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 10 monclient: _renew_subs 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a01394e0 con 0x7f13a00b1730 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.580+0000 7f13a7b9a640 1 librados: init done 2024-03-25T19:40:59.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.581+0000 7f13a7b9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.581+0000 7f13a7b9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.581+0000 7f13a7b9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:40:59.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.581+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a01d7860 con 0x7f13a00b1730 2024-03-25T19:40:59.669 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.424 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.733 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:09.476 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.779 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:09.521 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.791+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f139c001220 con 0x7f13a00b1730 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.791+0000 7f13967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.791+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f139c003a40 con 0x7f13a00b1730 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.791+0000 7f13967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.791+0000 7f13967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.791+0000 7f13967fc640 10 monclient: dump: 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:40:59.791 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:40:59.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.791+0000 7f1397fff640 4 set_mon_vals no callback set 2024-03-25T19:40:59.810 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:09.558 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.812 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.549 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.822+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f139c019a10 con 0x7f13a00b1730 2024-03-25T19:40:59.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.823+0000 7f13a5135640 10 monclient: get_auth_request con 0x7f138406e110 auth_method 0 2024-03-25T19:40:59.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.823+0000 7f13967fc640 10 client.4215.objecter ms_handle_connect 0x7f138406e110 2024-03-25T19:40:59.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.827+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f139c0535c0 con 0x7f13a00b1730 2024-03-25T19:40:59.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.827+0000 7f13967fc640 10 client.4215.objecter ms_dispatch 0x7f13a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:59.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.827+0000 7f13967fc640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:59.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.827+0000 7f13967fc640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:59.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.827+0000 7f13967fc640 20 client.4215.objecter dump_active .. 0 homeless 2024-03-25T19:40:59.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.827+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f139c04ef00 con 0x7f13a00b1730 2024-03-25T19:40:59.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.827+0000 7f13967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:40:59.827+0000 7f13967fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:59.832 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.563 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.843 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:09.565 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.882 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.613 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:59.963 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.718 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.005+0000 7f13a7b9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:00.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.005+0000 7f13a7b9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:00.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.005+0000 7f13a7b9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:00.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.005+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13a01401f0 con 0x7f13a00b1730 2024-03-25T19:41:00.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.019+0000 7f13967fc640 1 -- 172.21.15.69:0/3641168912 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f139c051000 con 0x7f13a00b1730 2024-03-25T19:41:00.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.019+0000 7f13967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:00.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.019+0000 7f13967fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:00.020 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:41:00.020 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:00.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 10 librados: watch_flush enter 2024-03-25T19:41:00.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 10 librados: watch_flush exit 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 20 client.4215.objecter shutdown clearing up homeless session... 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 10 client.4215.objecter successfully canceled tick 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 >> v1:172.21.15.169:6800/3072548259 conn(0x7f138406e110 legacy=0x7f13840705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 10 monclient: shutdown 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 >> v1:172.21.15.69:6789/0 conn(0x7f13a00b1730 legacy=0x7f13a01d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 shutdown_connections 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 >> 172.21.15.69:0/3641168912 conn(0x7f13a00617a0 msgr2=0x7f13a00b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 shutdown_connections 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 1 -- 172.21.15.69:0/3641168912 wait complete. 2024-03-25T19:41:00.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:00.021+0000 7f13a7b9a640 1 librados: shutdown 2024-03-25T19:41:00.023 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:09.764 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:00.053 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:09.795 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.079 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:09.827 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.103 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.839 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.110 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:09.832 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.132 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.863 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.175 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.907 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.250 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.005 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.315 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:10.058 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.321 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:10.063 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.360 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:10.108 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.382 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:10.104 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.399 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.136 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.429 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.161 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.471 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.203 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.472 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.203 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.541 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.296 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.586 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:10.328 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.609 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:10.352 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.660 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:10.382 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.691 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.428 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.760 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.491 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.828 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:10.576 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.837 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.593 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.881 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:10.629 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.904 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:10.647 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.932 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:10.654 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.984 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.715 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:00.984 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.721 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.037 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.769 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.050 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.782 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.062 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:10.805 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.114 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:10.857 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.125 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.881 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.348 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.079 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.440 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:11.162 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.463 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:11.206 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.499 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:11.222 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.503 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.240 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.519 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:11.261 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.558 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.294 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.659 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.414 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.713 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.469 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.945 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.676 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.010 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.742 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.351 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:16.099 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.464 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:16.206 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.837 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:06.853 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:16.575 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.943 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:16.686 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.978+0000 7f650a08f640 10 monclient: get_monmap_and_config 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.978+0000 7f650a08f640 10 monclient: build_initial_monmap 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.979+0000 7f650a08f640 10 monclient: monmap: 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:41:06.980203+0000 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:41:06.980203+0000 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:41:06.979 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.979+0000 7f650a08f640 1 Processor -- start 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.979+0000 7f650a08f640 1 -- start start 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.979+0000 7f650a08f640 10 monclient: init 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 10 monclient(hunting): picked mon.noname-a con 0x7f65040b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 10 monclient(hunting): picked mon.noname-b con 0x7f65040b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 10 monclient(hunting): picked mon.noname-c con 0x7f650414be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65040af940 con 0x7f65040b4d60 2024-03-25T19:41:06.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65040b0a70 con 0x7f650414be90 2024-03-25T19:41:06.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65040b0d50 con 0x7f65040b1730 2024-03-25T19:41:06.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:06.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f650a08f640 10 monclient(hunting): authenticate will time out at 4879.249178s 2024-03-25T19:41:06.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f6502ffd640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f65040b1730 0x7f65040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:33154/0 (socket says 172.21.15.69:33154) 2024-03-25T19:41:06.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.980+0000 7f6502ffd640 1 -- 172.21.15.69:0/2305546881 learned_addr learned my addr 172.21.15.69:0/2305546881 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.983+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833865637 0 0) 0x7f65040af940 con 0x7f65040b4d60 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.983+0000 7f65027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.983+0000 7f65027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.983+0000 7f65027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.983+0000 7f65027fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.983+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ec003520 con 0x7f65040b4d60 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.984+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214098691 0 0) 0x7f65040b0d50 con 0x7f65040b1730 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.984+0000 7f65027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.984+0000 7f65027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:06.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.984+0000 7f65027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:06.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.984+0000 7f65027fc640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:41:06.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.984+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65040af940 con 0x7f65040b1730 2024-03-25T19:41:06.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.985+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300093620 0 0) 0x7f65040b0a70 con 0x7f650414be90 2024-03-25T19:41:06.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.985+0000 7f65027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:06.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.985+0000 7f65027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:06.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.985+0000 7f65027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:06.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.985+0000 7f65027fc640 10 monclient(hunting): my global_id is 4201 2024-03-25T19:41:06.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.985+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65040b0d50 con 0x7f650414be90 2024-03-25T19:41:06.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 670210416 0 0) 0x7f65040af940 con 0x7f65040b1730 2024-03-25T19:41:06.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 >> v1:172.21.15.169:6789/0 conn(0x7f650414be90 legacy=0x7f650414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 >> v1:172.21.15.69:6789/0 conn(0x7f65040b4d60 legacy=0x7f650413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:06.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 1 monclient: found mon.noname-b 2024-03-25T19:41:06.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65040b1030 con 0x7f65040b1730 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f64f8001670 con 0x7f65040b1730 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.988+0000 7f65027fc640 10 monclient: dump: 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:06.990 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:06.991 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:06.991 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:06.991 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:06.991 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:06.991 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:06.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.989+0000 7f650a08f640 5 monclient: authenticate success, global_id 4205 2024-03-25T19:41:06.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.989+0000 7f650a08f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:06.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.991+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64f80017d0 con 0x7f65040b1730 2024-03-25T19:41:06.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.991+0000 7f65027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:06.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.991+0000 7f65027fc640 1 -- 172.21.15.69:0/2305546881 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f64f8001be0 con 0x7f65040b1730 2024-03-25T19:41:06.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.991+0000 7f65027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:06.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.991+0000 7f65027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:06.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.991+0000 7f65027fc640 10 monclient: dump: 2024-03-25T19:41:06.993 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:06.993 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.991+0000 7f650a08f640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.991+0000 7f650a08f640 4 set_mon_vals no callback set 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.992+0000 7f650a08f640 10 monclient: shutdown 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.992+0000 7f650a08f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.992+0000 7f650a08f640 1 -- 172.21.15.69:0/2305546881 >> v1:172.21.15.69:6790/0 conn(0x7f65040b1730 legacy=0x7f65040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.992+0000 7f650a08f640 1 -- 172.21.15.69:0/2305546881 shutdown_connections 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.992+0000 7f650a08f640 1 -- 172.21.15.69:0/2305546881 >> 172.21.15.69:0/2305546881 conn(0x7f65040617a0 msgr2=0x7f6504061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.992+0000 7f650a08f640 1 -- 172.21.15.69:0/2305546881 shutdown_connections 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.992+0000 7f650a08f640 1 -- 172.21.15.69:0/2305546881 wait complete. 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.993+0000 7f650a08f640 10 monclient: build_initial_monmap 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.993+0000 7f650a08f640 10 monclient: monmap: 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:41:06.995 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:06.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.993+0000 7f650a08f640 1 librados: starting msgr at 2024-03-25T19:41:06.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.993+0000 7f650a08f640 1 librados: starting objecter 2024-03-25T19:41:06.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.994+0000 7f650a08f640 1 Processor -- start 2024-03-25T19:41:06.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.994+0000 7f650a08f640 1 -- start start 2024-03-25T19:41:06.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.994+0000 7f650a08f640 1 librados: setting wanted keys 2024-03-25T19:41:06.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.994+0000 7f650a08f640 1 librados: calling monclient init 2024-03-25T19:41:06.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.994+0000 7f650a08f640 10 monclient: init 2024-03-25T19:41:06.997 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:16.726 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.995+0000 7f650a08f640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.995+0000 7f650a08f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.995+0000 7f650a08f640 10 monclient(hunting): picked mon.noname-a con 0x7f65040b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.996+0000 7f65037fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f65040b1730 0x7f6504135fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35366/0 (socket says 172.21.15.69:35366) 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.996+0000 7f650a08f640 10 monclient(hunting): picked mon.noname-c con 0x7f65040b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.996+0000 7f650a08f640 10 monclient(hunting): picked mon.noname-b con 0x7f650414be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.996+0000 7f650a08f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65041381d0 con 0x7f65040b1730 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.996+0000 7f65037fe640 1 -- 172.21.15.69:0/1212301081 learned_addr learned my addr 172.21.15.69:0/1212301081 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.997+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65041397e0 con 0x7f650414be90 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.997+0000 7f64ebfff640 10 client.?.objecter ms_handle_connect 0x7f65040b4d60 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.997+0000 7f64ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:07.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.997+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6504137cd0 con 0x7f65040b4d60 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.997+0000 7f650a08f640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.997+0000 7f650a08f640 10 monclient(hunting): authenticate will time out at 4879.266321s 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.997+0000 7f64ebfff640 10 client.?.objecter ms_handle_connect 0x7f65040b1730 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.997+0000 7f64ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.998+0000 7f64ebfff640 10 client.?.objecter ms_handle_connect 0x7f650414be90 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.998+0000 7f64ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.999+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965408939 0 0) 0x7f6504137cd0 con 0x7f65040b4d60 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.999+0000 7f64ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.999+0000 7f64ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.999+0000 7f64ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.999+0000 7f64ebfff640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:41:07.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:06.999+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f0004eb0 con 0x7f65040b4d60 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.000+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034310967 0 0) 0x7f65041397e0 con 0x7f650414be90 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.000+0000 7f64ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.000+0000 7f64ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.000+0000 7f64ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.000+0000 7f64ebfff640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.000+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6504137cd0 con 0x7f650414be90 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.001+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066641024 0 0) 0x7f65041381d0 con 0x7f65040b1730 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.001+0000 7f64ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.001+0000 7f64ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.001+0000 7f64ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:07.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.001+0000 7f64ebfff640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:41:07.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.001+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65041397e0 con 0x7f65040b1730 2024-03-25T19:41:07.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.006+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3705871817 0 0) 0x7f64f0004eb0 con 0x7f65040b4d60 2024-03-25T19:41:07.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.006+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65041381d0 con 0x7f65040b4d60 2024-03-25T19:41:07.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.006+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f64f8001ba0 con 0x7f65040b4d60 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.006+0000 7f64ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.006+0000 7f64ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.006+0000 7f64ebfff640 10 monclient(hunting): dump: 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:07.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.006+0000 7f64ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2395188594 0 0) 0x7f65041397e0 con 0x7f65040b1730 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64f0004eb0 con 0x7f65040b1730 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f64f4001910 con 0x7f65040b1730 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 10 monclient(hunting): dump: 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:07.009 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2472903947 0 0) 0x7f6504137cd0 con 0x7f650414be90 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65041397e0 con 0x7f650414be90 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f64fc001b50 con 0x7f650414be90 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.008+0000 7f64ebfff640 10 monclient(hunting): dump: 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:07.010 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:07.011 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:07.011 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:07.011 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:07.011 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:07.011 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:07.011 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:07.011 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:07.011 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 410420778 0 0) 0x7f65041381d0 con 0x7f65040b4d60 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 >> v1:172.21.15.169:6789/0 conn(0x7f650414be90 legacy=0x7f65041e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 >> v1:172.21.15.69:6789/0 conn(0x7f65040b1730 legacy=0x7f6504135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 1 monclient: found mon.c 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6504137e60 con 0x7f65040b4d60 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:07.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f64ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 5 monclient: authenticate success, global_id 4208 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 10 monclient: _renew_subs 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6504138df0 con 0x7f65040b4d60 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 10 monclient: _renew_subs 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6504145130 con 0x7f65040b4d60 2024-03-25T19:41:07.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.011+0000 7f650a08f640 1 librados: init done 2024-03-25T19:41:07.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.012+0000 7f650a08f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:07.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.012+0000 7f650a08f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:07.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.012+0000 7f650a08f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:07.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.012+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65041401f0 con 0x7f65040b4d60 2024-03-25T19:41:07.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.014+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64f8004a90 con 0x7f65040b4d60 2024-03-25T19:41:07.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.014+0000 7f64ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:07.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.014+0000 7f6501ffb640 4 set_mon_vals no callback set 2024-03-25T19:41:07.209 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:16.945 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.261+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f64f8004da0 con 0x7f65040b4d60 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.262+0000 7f64ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.262+0000 7f64ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.262+0000 7f64ebfff640 10 monclient: dump: 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:07.262 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:07.263 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:07.263 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:07.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.262+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f64f801ad70 con 0x7f65040b4d60 2024-03-25T19:41:07.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.263+0000 7f65037fe640 10 monclient: get_auth_request con 0x7f64f006e0f0 auth_method 0 2024-03-25T19:41:07.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.263+0000 7f64ebfff640 10 client.4208.objecter ms_handle_connect 0x7f64f006e0f0 2024-03-25T19:41:07.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.270+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f64f8054960 con 0x7f65040b4d60 2024-03-25T19:41:07.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.270+0000 7f64ebfff640 10 client.4208.objecter ms_dispatch 0x7f6504000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:07.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.270+0000 7f64ebfff640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:07.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.270+0000 7f64ebfff640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:07.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.270+0000 7f64ebfff640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T19:41:07.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.270+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64f80502a0 con 0x7f65040b4d60 2024-03-25T19:41:07.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.270+0000 7f64ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:07.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.270+0000 7f64ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:07.287 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:17.042 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.451+0000 7f650a08f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:07.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.451+0000 7f650a08f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:07.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.451+0000 7f650a08f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:07.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.451+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6504138360 con 0x7f65040b4d60 2024-03-25T19:41:07.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.475+0000 7f64ebfff640 1 -- 172.21.15.69:0/1212301081 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f64f80523a0 con 0x7f65040b4d60 2024-03-25T19:41:07.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.475+0000 7f64ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:07.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.475+0000 7f64ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:07.476 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:41:07.476 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 10 librados: watch_flush enter 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 10 librados: watch_flush exit 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 10 client.4208.objecter successfully canceled tick 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 >> v1:172.21.15.169:6800/3072548259 conn(0x7f64f006e0f0 legacy=0x7f64f00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 10 monclient: shutdown 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 >> v1:172.21.15.69:6790/0 conn(0x7f65040b4d60 legacy=0x7f65041366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 shutdown_connections 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.478+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 >> 172.21.15.69:0/1212301081 conn(0x7f65040617a0 msgr2=0x7f65040b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.479+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 shutdown_connections 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.479+0000 7f650a08f640 1 -- 172.21.15.69:0/1212301081 wait complete. 2024-03-25T19:41:07.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:07.479+0000 7f650a08f640 1 librados: shutdown 2024-03-25T19:41:07.491 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:07.513 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:17.244 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.824 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:21.572 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.830 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:21.578 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.980 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:21.722 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.984 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:21.726 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.179 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:21.901 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.184 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:21.907 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.344 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:22.086 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.348 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:22.091 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.809 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:22.540 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.812 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:22.544 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.956 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:22.693 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.962 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:22.699 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.964 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:22.696 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.968 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:22.700 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.988 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:22.744 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.994 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:22.749 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:14.298 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.399+0000 7fc9086ba640 10 monclient: get_monmap_and_config 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.399+0000 7fc9086ba640 10 monclient: build_initial_monmap 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.399+0000 7fc9086ba640 10 monclient: monmap: 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:41:14.400258+0000 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:41:14.400258+0000 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:14.399 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:14.400 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:14.400 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:41:14.400 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:41:14.400 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:14.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.399+0000 7fc9086ba640 1 Processor -- start 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.399+0000 7fc9086ba640 1 -- start start 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.399+0000 7fc9086ba640 10 monclient: init 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.400+0000 7fc9086ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.400+0000 7fc9086ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc9086ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc9000a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc9086ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc90013f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc9086ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc9001430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc9086ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900142520 con 0x7fc9000a98f0 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc9086ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b28a0 con 0x7fc9001430c0 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc9086ba640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b2a30 con 0x7fc90013f710 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc9086ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc9086ba640 10 monclient(hunting): authenticate will time out at 4886.669824s 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc906c57640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc9001430c0 0x7fc90014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:51902/0 (socket says 172.21.15.69:51902) 2024-03-25T19:41:14.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.401+0000 7fc906c57640 1 -- 172.21.15.69:0/1694072956 learned_addr learned my addr 172.21.15.69:0/1694072956 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:14.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.463+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131426454 0 0) 0x7fc900142520 con 0x7fc9000a98f0 2024-03-25T19:41:14.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.463+0000 7fc905454640 10 monclient(hunting): _negotiate 2024-03-25T19:41:14.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.463+0000 7fc905454640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:14.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.463+0000 7fc905454640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:14.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.463+0000 7fc905454640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:41:14.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.463+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec003520 con 0x7fc9000a98f0 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.467+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727465970 0 0) 0x7fc8ec003520 con 0x7fc9000a98f0 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.467+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 >> v1:172.21.15.69:6790/0 conn(0x7fc90013f710 legacy=0x7fc9001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.467+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 >> v1:172.21.15.169:6789/0 conn(0x7fc9001430c0 legacy=0x7fc90014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 10 monclient: _finish_hunting 0 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 1 monclient: found mon.noname-a 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000b44c0 con 0x7fc9000a98f0 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 10 monclient: _finish_auth 0 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 10 monclient: _check_auth_tickets 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:14.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc9086ba640 5 monclient: authenticate success, global_id 4224 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc9086ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc8fc001690 con 0x7fc9000a98f0 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc905454640 10 monclient: dump: 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:14.469 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:14.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.468+0000 7fc9086ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8fc001920 con 0x7fc9000a98f0 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc905454640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc9086ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc9086ba640 4 set_mon_vals no callback set 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc905454640 1 -- 172.21.15.69:0/1694072956 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc8fc001690 con 0x7fc9000a98f0 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc905454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc905454640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc905454640 10 monclient: dump: 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:14.471 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:14.472 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:14.472 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.472 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:14.472 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:14.472 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:14.472 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:14.472 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:14.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.471+0000 7fc9086ba640 10 monclient: shutdown 2024-03-25T19:41:14.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.472+0000 7fc9086ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:14.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.472+0000 7fc9086ba640 1 -- 172.21.15.69:0/1694072956 >> v1:172.21.15.69:6789/0 conn(0x7fc9000a98f0 legacy=0x7fc9000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.472+0000 7fc9086ba640 1 -- 172.21.15.69:0/1694072956 shutdown_connections 2024-03-25T19:41:14.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.472+0000 7fc9086ba640 1 -- 172.21.15.69:0/1694072956 >> 172.21.15.69:0/1694072956 conn(0x7fc9000b7ea0 msgr2=0x7fc9000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:14.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.472+0000 7fc9086ba640 1 -- 172.21.15.69:0/1694072956 shutdown_connections 2024-03-25T19:41:14.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.472+0000 7fc9086ba640 1 -- 172.21.15.69:0/1694072956 wait complete. 2024-03-25T19:41:14.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.473+0000 7fc9086ba640 10 monclient: build_initial_monmap 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.473+0000 7fc9086ba640 10 monclient: monmap: 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.474+0000 7fc9086ba640 1 librados: starting msgr at 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.474+0000 7fc9086ba640 1 librados: starting objecter 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.474+0000 7fc9086ba640 1 Processor -- start 2024-03-25T19:41:14.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.474+0000 7fc9086ba640 1 -- start start 2024-03-25T19:41:14.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.474+0000 7fc9086ba640 1 librados: setting wanted keys 2024-03-25T19:41:14.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.474+0000 7fc9086ba640 1 librados: calling monclient init 2024-03-25T19:41:14.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.474+0000 7fc9086ba640 10 monclient: init 2024-03-25T19:41:14.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.475+0000 7fc9086ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:14.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.475+0000 7fc9086ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:14.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.475+0000 7fc9086ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc9000a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:14.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.475+0000 7fc9086ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc90013f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.476+0000 7fc9086ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc9001430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.476+0000 7fc9086ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9001377d0 con 0x7fc9001430c0 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.476+0000 7fc9086ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9001389f0 con 0x7fc90013f710 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.476+0000 7fc9086ba640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900136590 con 0x7fc9000a98f0 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.476+0000 7fc9086ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.476+0000 7fc9086ba640 10 monclient(hunting): authenticate will time out at 4886.745066s 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.476+0000 7fc906c57640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc9001430c0 0x7fc900132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45904/0 (socket says 172.21.15.69:45904) 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.476+0000 7fc906c57640 1 -- 172.21.15.69:0/2639389916 learned_addr learned my addr 172.21.15.69:0/2639389916 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.477+0000 7fc8f6ffd640 10 client.?.objecter ms_handle_connect 0x7fc9001430c0 2024-03-25T19:41:14.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.477+0000 7fc8f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:14.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.480+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192289826 0 0) 0x7fc9001377d0 con 0x7fc9001430c0 2024-03-25T19:41:14.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.480+0000 7fc8f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:14.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.480+0000 7fc8f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:14.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.480+0000 7fc8f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:14.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.480+0000 7fc8f6ffd640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:41:14.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.480+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e4004eb0 con 0x7fc9001430c0 2024-03-25T19:41:14.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.487+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 728949031 0 0) 0x7fc8e4004eb0 con 0x7fc9001430c0 2024-03-25T19:41:14.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.487+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9001377d0 con 0x7fc9001430c0 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.488+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc8f8001c70 con 0x7fc9001430c0 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.488+0000 7fc8f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.488+0000 7fc8f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.488+0000 7fc8f6ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:14.488 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:14.489 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:14.489 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:14.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.488+0000 7fc8f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:14.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1751568388 0 0) 0x7fc9001377d0 con 0x7fc9001430c0 2024-03-25T19:41:14.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 >> v1:172.21.15.69:6790/0 conn(0x7fc9000a98f0 legacy=0x7fc9001370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 >> v1:172.21.15.169:6789/0 conn(0x7fc90013f710 legacy=0x7fc900139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:14.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 1 monclient: found mon.a 2024-03-25T19:41:14.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:14.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:14.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900139790 con 0x7fc9001430c0 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc8f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 5 monclient: authenticate success, global_id 4227 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 10 monclient: _renew_subs 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9001399e0 con 0x7fc9001430c0 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 10 monclient: _renew_subs 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90013b530 con 0x7fc9001430c0 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.494+0000 7fc9086ba640 1 librados: init done 2024-03-25T19:41:14.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.495+0000 7fc9086ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:14.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.496+0000 7fc9086ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:14.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.496+0000 7fc9086ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:14.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.496+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900136770 con 0x7fc9001430c0 2024-03-25T19:41:14.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.838+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8f8004b60 con 0x7fc9001430c0 2024-03-25T19:41:14.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.838+0000 7fc8f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:14.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.838+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc8f8004e10 con 0x7fc9001430c0 2024-03-25T19:41:14.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.838+0000 7fc8f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.838+0000 7fc8f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:14.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.838+0000 7fc8f6ffd640 10 monclient: dump: 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.838+0000 7fc904c53640 4 set_mon_vals no callback set 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.838+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fc8f8005030 con 0x7fc9001430c0 2024-03-25T19:41:14.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.839+0000 7fc906456640 10 monclient: get_auth_request con 0x7fc8e406dbc0 auth_method 0 2024-03-25T19:41:14.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.839+0000 7fc8f6ffd640 10 client.4227.objecter ms_handle_connect 0x7fc8e406dbc0 2024-03-25T19:41:14.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.845+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fc8f8054a80 con 0x7fc9001430c0 2024-03-25T19:41:14.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.845+0000 7fc8f6ffd640 10 client.4227.objecter ms_dispatch 0x7fc900000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:14.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.845+0000 7fc8f6ffd640 3 client.4227.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:14.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.845+0000 7fc8f6ffd640 3 client.4227.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:14.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.845+0000 7fc8f6ffd640 20 client.4227.objecter dump_active .. 0 homeless 2024-03-25T19:41:14.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.846+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8f8050590 con 0x7fc9001430c0 2024-03-25T19:41:14.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.846+0000 7fc8f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:14.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:14.846+0000 7fc8f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:15.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.028+0000 7fc9086ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:15.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.028+0000 7fc9086ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:15.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.028+0000 7fc9086ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:15.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.028+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9000627b0 con 0x7fc9001430c0 2024-03-25T19:41:15.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.044+0000 7fc8f6ffd640 1 -- 172.21.15.69:0/2639389916 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fc8f80524c0 con 0x7fc9001430c0 2024-03-25T19:41:15.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.044+0000 7fc8f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:15.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.044+0000 7fc8f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:15.044 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:41:15.044 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:15.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 10 librados: watch_flush enter 2024-03-25T19:41:15.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 10 librados: watch_flush exit 2024-03-25T19:41:15.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 20 client.4227.objecter shutdown clearing up homeless session... 2024-03-25T19:41:15.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 10 client.4227.objecter successfully canceled tick 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 >> v1:172.21.15.169:6800/3072548259 conn(0x7fc8e406dbc0 legacy=0x7fc8e4070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 10 monclient: shutdown 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 >> v1:172.21.15.69:6789/0 conn(0x7fc9001430c0 legacy=0x7fc900132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 shutdown_connections 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 >> 172.21.15.69:0/2639389916 conn(0x7fc9000b7ea0 msgr2=0x7fc90013d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 shutdown_connections 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 1 -- 172.21.15.69:0/2639389916 wait complete. 2024-03-25T19:41:15.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:15.046+0000 7fc9086ba640 1 librados: shutdown 2024-03-25T19:41:15.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:16.193 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:25.935 35592== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:16.344 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:26.092 35590== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:16.636 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:26.358 35597== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:16.729 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:26.471 35591== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.209 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:26.964 35694== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.355 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:27.092 35695== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.493 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:27.225 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.568 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:27.299 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:21.862 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:22.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.005+0000 7f9ac49c2640 10 monclient: get_monmap_and_config 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.005+0000 7f9ac49c2640 10 monclient: build_initial_monmap 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.005+0000 7f9ac49c2640 10 monclient: monmap: 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:41:22.005935+0000 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:41:22.005935+0000 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:22.005 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:41:22.006 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:41:22.006 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:22.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.006+0000 7f9ac49c2640 1 Processor -- start 2024-03-25T19:41:22.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.006+0000 7f9ac49c2640 1 -- start start 2024-03-25T19:41:22.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.006+0000 7f9ac49c2640 10 monclient: init 2024-03-25T19:41:22.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:22.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:22.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:22.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac00b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:22.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac014be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:22.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00af940 con 0x7f9ac014be90 2024-03-25T19:41:22.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00b0a70 con 0x7f9ac00b4d60 2024-03-25T19:41:22.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00b0d50 con 0x7f9ac00b1730 2024-03-25T19:41:22.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:22.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.007+0000 7f9ac49c2640 10 monclient(hunting): authenticate will time out at 4894.275523s 2024-03-25T19:41:22.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.008+0000 7f9ab7fff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f9ac00b1730 0x7f9ac00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:38452/0 (socket says 172.21.15.69:38452) 2024-03-25T19:41:22.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.008+0000 7f9ab7fff640 1 -- 172.21.15.69:0/649659056 learned_addr learned my addr 172.21.15.69:0/649659056 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:22.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.011+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391099809 0 0) 0x7f9ac00b0a70 con 0x7f9ac00b4d60 2024-03-25T19:41:22.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.011+0000 7f9abdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:22.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.011+0000 7f9abdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:22.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.011+0000 7f9abdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:22.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.011+0000 7f9abdd9b640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:41:22.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.011+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac003520 con 0x7f9ac00b4d60 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.012+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671887376 0 0) 0x7f9ac00af940 con 0x7f9ac014be90 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.012+0000 7f9abdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.012+0000 7f9abdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.012+0000 7f9abdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.012+0000 7f9abdd9b640 10 monclient(hunting): my global_id is 4230 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.012+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac00b0a70 con 0x7f9ac014be90 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.013+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509956561 0 0) 0x7f9ac00b0d50 con 0x7f9ac00b1730 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.013+0000 7f9abdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.013+0000 7f9abdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.013+0000 7f9abdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.013+0000 7f9abdd9b640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:41:22.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.013+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac00af940 con 0x7f9ac00b1730 2024-03-25T19:41:22.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1726929446 0 0) 0x7f9ac00af940 con 0x7f9ac00b1730 2024-03-25T19:41:22.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 >> v1:172.21.15.169:6789/0 conn(0x7f9ac00b4d60 legacy=0x7f9ac013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 >> v1:172.21.15.69:6789/0 conn(0x7f9ac014be90 legacy=0x7f9ac014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:22.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 1 monclient: found mon.noname-b 2024-03-25T19:41:22.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac00b1030 con 0x7f9ac00b1730 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9aa80015b0 con 0x7f9ac00b1730 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9abdd9b640 10 monclient: dump: 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.018 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:22.019 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:22.019 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:22.019 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:22.019 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:22.019 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:22.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9ac49c2640 5 monclient: authenticate success, global_id 4214 2024-03-25T19:41:22.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.017+0000 7f9ac49c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.020+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa80017d0 con 0x7f9ac00b1730 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.020+0000 7f9abdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.020+0000 7f9abdd9b640 1 -- 172.21.15.69:0/649659056 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9aa8001b60 con 0x7f9ac00b1730 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.020+0000 7f9abdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.020+0000 7f9abdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.020+0000 7f9abdd9b640 10 monclient: dump: 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:22.021 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.020+0000 7f9ac49c2640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.020+0000 7f9ac49c2640 4 set_mon_vals no callback set 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.021+0000 7f9ac49c2640 10 monclient: shutdown 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.021+0000 7f9ac49c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.021+0000 7f9ac49c2640 1 -- 172.21.15.69:0/649659056 >> v1:172.21.15.69:6790/0 conn(0x7f9ac00b1730 legacy=0x7f9ac00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.021+0000 7f9ac49c2640 1 -- 172.21.15.69:0/649659056 shutdown_connections 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.021+0000 7f9ac49c2640 1 -- 172.21.15.69:0/649659056 >> 172.21.15.69:0/649659056 conn(0x7f9ac00617a0 msgr2=0x7f9ac0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.022+0000 7f9ac49c2640 1 -- 172.21.15.69:0/649659056 shutdown_connections 2024-03-25T19:41:22.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.022+0000 7f9ac49c2640 1 -- 172.21.15.69:0/649659056 wait complete. 2024-03-25T19:41:22.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 10 monclient: build_initial_monmap 2024-03-25T19:41:22.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 10 monclient: monmap: 2024-03-25T19:41:22.024 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:22.024 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:22.024 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:41:22.024 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 1 librados: starting msgr at 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 1 librados: starting objecter 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 1 Processor -- start 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 1 -- start start 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 1 librados: setting wanted keys 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 1 librados: calling monclient init 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.023+0000 7f9ac49c2640 10 monclient: init 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.024+0000 7f9ac49c2640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:22.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.024+0000 7f9ac49c2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.024+0000 7f9ac49c2640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ac49c2640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac00b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ac49c2640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ac49c2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac01db770 con 0x7f9ac00b4d60 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ac49c2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac0137710 con 0x7f9ac00b1730 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ac49c2640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac01378a0 con 0x7f9ac014be90 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ac49c2640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ab7fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9ac00b4d60 0x7f9ac0137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41568/0 (socket says 172.21.15.69:41568) 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ab7fff640 1 -- 172.21.15.69:0/1701170410 learned_addr learned my addr 172.21.15.69:0/1701170410 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:22.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ac49c2640 10 monclient(hunting): authenticate will time out at 4894.293145s 2024-03-25T19:41:22.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.025+0000 7f9ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f9ac00b4d60 2024-03-25T19:41:22.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.026+0000 7f9ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:22.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.026+0000 7f9ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f9ac00b1730 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.026+0000 7f9ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.028+0000 7f9ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f9ac014be90 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.028+0000 7f9ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.029+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626675001 0 0) 0x7f9ac01db770 con 0x7f9ac00b4d60 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.029+0000 7f9ab6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.029+0000 7f9ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.029+0000 7f9ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.029+0000 7f9ab6ffd640 10 monclient(hunting): my global_id is 4233 2024-03-25T19:41:22.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.029+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa0004eb0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.030+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273145004 0 0) 0x7f9ac0137710 con 0x7f9ac00b1730 2024-03-25T19:41:22.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.030+0000 7f9ab6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:22.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.030+0000 7f9ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:22.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.030+0000 7f9ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:22.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.030+0000 7f9ab6ffd640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:41:22.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.030+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac01db770 con 0x7f9ac00b1730 2024-03-25T19:41:22.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.032+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064508491 0 0) 0x7f9ac01378a0 con 0x7f9ac014be90 2024-03-25T19:41:22.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.032+0000 7f9ab6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:22.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.032+0000 7f9ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:22.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.032+0000 7f9ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:22.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.032+0000 7f9ab6ffd640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:41:22.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.032+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa0005360 con 0x7f9ac014be90 2024-03-25T19:41:22.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.037+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1867509919 0 0) 0x7f9aa0004eb0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.037+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ac01378a0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.037+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9aa80015b0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.037+0000 7f9ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:22.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.037+0000 7f9ab6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:22.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.037+0000 7f9ab6ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:22.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.037+0000 7f9ab6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1394588988 0 0) 0x7f9ac01db770 con 0x7f9ac00b1730 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aa0004eb0 con 0x7f9ac00b1730 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9ab00018d0 con 0x7f9ac00b1730 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:22.039 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1788115263 0 0) 0x7f9aa0005360 con 0x7f9ac014be90 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aa0006150 con 0x7f9ac014be90 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9ab8001b00 con 0x7f9ac014be90 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.038+0000 7f9ab6ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:22.040 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:22.041 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:22.041 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:22.041 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:22.041 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:22.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1349973717 0 0) 0x7f9ac01378a0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 >> v1:172.21.15.69:6790/0 conn(0x7f9ac014be90 legacy=0x7f9ac0141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 >> v1:172.21.15.169:6789/0 conn(0x7f9ac00b1730 legacy=0x7f9ac01daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:22.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 1 monclient: found mon.a 2024-03-25T19:41:22.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac01db1c0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ab6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 5 monclient: authenticate success, global_id 4233 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 10 monclient: _renew_subs 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac01db470 con 0x7f9ac00b4d60 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 10 monclient: _renew_subs 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac013aef0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.043+0000 7f9ac49c2640 1 librados: init done 2024-03-25T19:41:22.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.045+0000 7f9ac49c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:22.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.045+0000 7f9ac49c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:22.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.045+0000 7f9ac49c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:22.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.045+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac00627b0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.045+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa8004a90 con 0x7f9ac00b4d60 2024-03-25T19:41:22.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.045+0000 7f9ab6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:22.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.045+0000 7f9abd59a640 4 set_mon_vals no callback set 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.047+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9aa8004da0 con 0x7f9ac00b4d60 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.047+0000 7f9ab6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.047+0000 7f9ab6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.047+0000 7f9ab6ffd640 10 monclient: dump: 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:22.048 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:22.071 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.072+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f9aa801ad70 con 0x7f9ac00b4d60 2024-03-25T19:41:22.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.073+0000 7f9abe59c640 10 monclient: get_auth_request con 0x7f9aa006e5f0 auth_method 0 2024-03-25T19:41:22.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.074+0000 7f9ab6ffd640 10 client.4233.objecter ms_handle_connect 0x7f9aa006e5f0 2024-03-25T19:41:22.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.392+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f9aa8054900 con 0x7f9ac00b4d60 2024-03-25T19:41:22.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.392+0000 7f9ab6ffd640 10 client.4233.objecter ms_dispatch 0x7f9ac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:22.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.392+0000 7f9ab6ffd640 3 client.4233.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:22.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.392+0000 7f9ab6ffd640 3 client.4233.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:22.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.393+0000 7f9ab6ffd640 20 client.4233.objecter dump_active .. 0 homeless 2024-03-25T19:41:22.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.393+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9aa8050240 con 0x7f9ac00b4d60 2024-03-25T19:41:22.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.393+0000 7f9ab6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:22.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.393+0000 7f9ab6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:22.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.665+0000 7f9ac49c2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:22.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.665+0000 7f9ac49c2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:22.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.665+0000 7f9ac49c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:22.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.665+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ac0069f40 con 0x7f9ac00b4d60 2024-03-25T19:41:22.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.681+0000 7f9ab6ffd640 1 -- 172.21.15.69:0/1701170410 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f9aa8052340 con 0x7f9ac00b4d60 2024-03-25T19:41:22.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.682+0000 7f9ab6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:22.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.682+0000 7f9ab6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:22.681 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:41:22.681 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.683+0000 7f9ac49c2640 10 librados: watch_flush enter 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.683+0000 7f9ac49c2640 10 librados: watch_flush exit 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.683+0000 7f9ac49c2640 20 client.4233.objecter shutdown clearing up homeless session... 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.683+0000 7f9ac49c2640 10 client.4233.objecter successfully canceled tick 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.683+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 >> v1:172.21.15.169:6800/3072548259 conn(0x7f9aa006e5f0 legacy=0x7f9aa0070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.684+0000 7f9ac49c2640 10 monclient: shutdown 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.684+0000 7f9ac49c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.684+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 >> v1:172.21.15.69:6789/0 conn(0x7f9ac00b4d60 legacy=0x7f9ac0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.684+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 shutdown_connections 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.684+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 >> 172.21.15.69:0/1701170410 conn(0x7f9ac00617a0 msgr2=0x7f9ac0145390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.684+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 shutdown_connections 2024-03-25T19:41:22.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.684+0000 7f9ac49c2640 1 -- 172.21.15.69:0/1701170410 wait complete. 2024-03-25T19:41:22.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:22.684+0000 7f9ac49c2640 1 librados: shutdown 2024-03-25T19:41:22.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:24.784 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-25T19:41:24.781+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:41:24.795 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-25T19:41:24.793+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:41:25.079 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-25T19:41:25.076+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:41:25.508 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-25T19:41:25.506+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:41:26.147 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-03-25T19:41:26.141+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:41:26.212 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-03-25T19:41:26.209+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:41:26.492 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-03-25T19:41:26.390+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:41:26.872 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-03-25T19:41:26.868+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:41:29.500 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.606+0000 7fe2cfce4640 10 monclient: get_monmap_and_config 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.606+0000 7fe2cfce4640 10 monclient: build_initial_monmap 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.606+0000 7fe2cfce4640 10 monclient: monmap: 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:41:29.608119+0000 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:41:29.608119+0000 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:29.607 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:41:29.608 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:41:29.608 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:29.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.607+0000 7fe2cfce4640 1 Processor -- start 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.607+0000 7fe2cfce4640 1 -- start start 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.607+0000 7fe2cfce4640 10 monclient: init 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c80b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c80b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c814be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80af940 con 0x7fe2c80b4d60 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80b0a70 con 0x7fe2c80b1730 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80b0d50 con 0x7fe2c814be90 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:29.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.608+0000 7fe2cfce4640 10 monclient(hunting): authenticate will time out at 4901.877210s 2024-03-25T19:41:29.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.621+0000 7fe2cda80640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe2c80b4d60 0x7fe2c813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42752/0 (socket says 172.21.15.69:42752) 2024-03-25T19:41:29.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.621+0000 7fe2cda80640 1 -- 172.21.15.69:0/1515865862 learned_addr learned my addr 172.21.15.69:0/1515865862 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:29.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.630+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565614912 0 0) 0x7fe2c80b0a70 con 0x7fe2c80b1730 2024-03-25T19:41:29.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.630+0000 7fe2cca7e640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.630+0000 7fe2cca7e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.630+0000 7fe2cca7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.630+0000 7fe2cca7e640 10 monclient(hunting): my global_id is 4237 2024-03-25T19:41:29.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.630+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b4003520 con 0x7fe2c80b1730 2024-03-25T19:41:29.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.640+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945284 0 0) 0x7fe2c80af940 con 0x7fe2c80b4d60 2024-03-25T19:41:29.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.640+0000 7fe2cca7e640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.640+0000 7fe2cca7e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.640+0000 7fe2cca7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.640+0000 7fe2cca7e640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:41:29.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.640+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c80b0a70 con 0x7fe2c80b4d60 2024-03-25T19:41:29.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 234494744 0 0) 0x7fe2c80b0a70 con 0x7fe2c80b4d60 2024-03-25T19:41:29.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 >> v1:172.21.15.69:6790/0 conn(0x7fe2c814be90 legacy=0x7fe2c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 >> v1:172.21.15.169:6789/0 conn(0x7fe2c80b1730 legacy=0x7fe2c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 10 monclient: _finish_hunting 0 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 1 monclient: found mon.noname-a 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c80b1030 con 0x7fe2c80b4d60 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 10 monclient: _finish_auth 0 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 10 monclient: _check_auth_tickets 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe2c4001670 con 0x7fe2c80b4d60 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cfce4640 5 monclient: authenticate success, global_id 4260 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cfce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cca7e640 10 monclient: dump: 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:29.654 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:29.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.653+0000 7fe2cfce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:29.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.665+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2c40017d0 con 0x7fe2c80b4d60 2024-03-25T19:41:29.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.665+0000 7fe2cca7e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:29.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.665+0000 7fe2cca7e640 1 -- 172.21.15.69:0/1515865862 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe2c4001b60 con 0x7fe2c80b4d60 2024-03-25T19:41:29.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.665+0000 7fe2cfce4640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:29.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.665+0000 7fe2cfce4640 4 set_mon_vals no callback set 2024-03-25T19:41:29.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.665+0000 7fe2cca7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.665+0000 7fe2cca7e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.665+0000 7fe2cca7e640 10 monclient: dump: 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:29.666 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:29.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.666+0000 7fe2cfce4640 10 monclient: shutdown 2024-03-25T19:41:29.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.666+0000 7fe2cfce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:29.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.666+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1515865862 >> v1:172.21.15.69:6789/0 conn(0x7fe2c80b4d60 legacy=0x7fe2c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.666+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1515865862 shutdown_connections 2024-03-25T19:41:29.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.666+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1515865862 >> 172.21.15.69:0/1515865862 conn(0x7fe2c80617a0 msgr2=0x7fe2c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:29.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.666+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1515865862 shutdown_connections 2024-03-25T19:41:29.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.667+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1515865862 wait complete. 2024-03-25T19:41:29.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 10 monclient: build_initial_monmap 2024-03-25T19:41:29.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 10 monclient: monmap: 2024-03-25T19:41:29.668 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:29.668 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:29.668 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 1 librados: starting msgr at 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 1 librados: starting objecter 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 1 Processor -- start 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 1 -- start start 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 1 librados: setting wanted keys 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 1 librados: calling monclient init 2024-03-25T19:41:29.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.668+0000 7fe2cfce4640 10 monclient: init 2024-03-25T19:41:29.670 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:29.670 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:29.670 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c80b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:29.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c80b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:29.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c814be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:29.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c81dacc0 con 0x7fe2c814be90 2024-03-25T19:41:29.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c81dbb40 con 0x7fe2c80b4d60 2024-03-25T19:41:29.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80aabb0 con 0x7fe2c80b1730 2024-03-25T19:41:29.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.669+0000 7fe2cfce4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:29.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.670+0000 7fe2cfce4640 10 monclient(hunting): authenticate will time out at 4901.938614s 2024-03-25T19:41:29.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.680+0000 7fe2cd27f640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe2c80b4d60 0x7fe2c80a96e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:56026/0 (socket says 172.21.15.69:56026) 2024-03-25T19:41:29.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.680+0000 7fe2cd27f640 1 -- 172.21.15.69:0/1180581565 learned_addr learned my addr 172.21.15.69:0/1180581565 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:29.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.680+0000 7fe2be7fc640 10 client.?.objecter ms_handle_connect 0x7fe2c80b4d60 2024-03-25T19:41:29.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.680+0000 7fe2be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:29.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.687+0000 7fe2be7fc640 10 client.?.objecter ms_handle_connect 0x7fe2c814be90 2024-03-25T19:41:29.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.687+0000 7fe2be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:29.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.690+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181234231 0 0) 0x7fe2c81dbb40 con 0x7fe2c80b4d60 2024-03-25T19:41:29.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.690+0000 7fe2be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.690+0000 7fe2be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.690+0000 7fe2be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.690+0000 7fe2be7fc640 10 monclient(hunting): my global_id is 4240 2024-03-25T19:41:29.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.690+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ac004eb0 con 0x7fe2c80b4d60 2024-03-25T19:41:29.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.691+0000 7fe2be7fc640 10 client.?.objecter ms_handle_connect 0x7fe2c80b1730 2024-03-25T19:41:29.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.691+0000 7fe2be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:29.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.699+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661650526 0 0) 0x7fe2c80aabb0 con 0x7fe2c80b1730 2024-03-25T19:41:29.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.699+0000 7fe2be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.699+0000 7fe2be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.699+0000 7fe2be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.699+0000 7fe2be7fc640 10 monclient(hunting): my global_id is 4244 2024-03-25T19:41:29.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.699+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c81dbb40 con 0x7fe2c80b1730 2024-03-25T19:41:29.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.700+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080253472 0 0) 0x7fe2c81dacc0 con 0x7fe2c814be90 2024-03-25T19:41:29.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.700+0000 7fe2be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.700+0000 7fe2be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.700+0000 7fe2be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.700 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.700+0000 7fe2be7fc640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:41:29.700 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.700+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c80aabb0 con 0x7fe2c814be90 2024-03-25T19:41:29.701 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.701+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1978277532 0 0) 0x7fe2ac004eb0 con 0x7fe2c80b4d60 2024-03-25T19:41:29.701 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.701+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2c81dacc0 con 0x7fe2c80b4d60 2024-03-25T19:41:29.701 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.701+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe2b8001990 con 0x7fe2c80b4d60 2024-03-25T19:41:29.701 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.701+0000 7fe2be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.701 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.701+0000 7fe2be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.701+0000 7fe2be7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:29.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.701+0000 7fe2be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:29.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.716+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1574567448 0 0) 0x7fe2c80aabb0 con 0x7fe2c814be90 2024-03-25T19:41:29.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.716+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2ac004eb0 con 0x7fe2c814be90 2024-03-25T19:41:29.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.716+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe2c0001ad0 con 0x7fe2c814be90 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.716+0000 7fe2be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.716+0000 7fe2be7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.716+0000 7fe2be7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:29.717 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.722+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096289545 0 0) 0x7fe2c81dbb40 con 0x7fe2c80b1730 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.722+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2c80aabb0 con 0x7fe2c80b1730 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.722+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe2c4001ae0 con 0x7fe2c80b1730 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.722+0000 7fe2be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.722+0000 7fe2be7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.722+0000 7fe2be7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:29.722 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.723 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:29.723 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:29.723 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:29.723 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:29.723 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 201967718 0 0) 0x7fe2c81dacc0 con 0x7fe2c80b4d60 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 >> v1:172.21.15.69:6790/0 conn(0x7fe2c80b1730 legacy=0x7fe2c81db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 >> v1:172.21.15.69:6789/0 conn(0x7fe2c814be90 legacy=0x7fe2c8137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 1 monclient: found mon.b 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c81daf10 con 0x7fe2c80b4d60 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2cfce4640 5 monclient: authenticate success, global_id 4240 2024-03-25T19:41:29.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2cfce4640 10 monclient: _renew_subs 2024-03-25T19:41:29.732 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2cfce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:29.732 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.730+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c80a9df0 con 0x7fe2c80b4d60 2024-03-25T19:41:29.732 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.731+0000 7fe2cfce4640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:29.732 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.731+0000 7fe2cfce4640 10 monclient: _renew_subs 2024-03-25T19:41:29.732 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.731+0000 7fe2cfce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:29.732 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.731+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c813ba20 con 0x7fe2c80b4d60 2024-03-25T19:41:29.732 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.731+0000 7fe2cfce4640 1 librados: init done 2024-03-25T19:41:29.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.732+0000 7fe2cfce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:29.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.732+0000 7fe2cfce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:29.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.732+0000 7fe2cfce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:29.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:29.732+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c80627b0 con 0x7fe2c80b4d60 2024-03-25T19:41:30.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.033+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2b8003610 con 0x7fe2c80b4d60 2024-03-25T19:41:30.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.033+0000 7fe2be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:30.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.033+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe2b8003960 con 0x7fe2c80b4d60 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.033+0000 7fe2be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.033+0000 7fe2be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.033+0000 7fe2be7fc640 10 monclient: dump: 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:30.034 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:30.035 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:30.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.033+0000 7fe2bffff640 4 set_mon_vals no callback set 2024-03-25T19:41:30.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.034+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fe2b8003b80 con 0x7fe2c80b4d60 2024-03-25T19:41:30.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.035+0000 7fe2cda80640 10 monclient: get_auth_request con 0x7fe2ac06e0f0 auth_method 0 2024-03-25T19:41:30.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.035+0000 7fe2be7fc640 10 client.4240.objecter ms_handle_connect 0x7fe2ac06e0f0 2024-03-25T19:41:30.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.039+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fe2b80546f0 con 0x7fe2c80b4d60 2024-03-25T19:41:30.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.039+0000 7fe2be7fc640 10 client.4240.objecter ms_dispatch 0x7fe2c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:30.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.039+0000 7fe2be7fc640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:30.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.039+0000 7fe2be7fc640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:30.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.039+0000 7fe2be7fc640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-25T19:41:30.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.040+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2b8050200 con 0x7fe2c80b4d60 2024-03-25T19:41:30.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.040+0000 7fe2be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:30.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.040+0000 7fe2be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:30.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.292+0000 7fe2cfce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:30.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.292+0000 7fe2cfce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:30.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.292+0000 7fe2cfce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:30.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.292+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2c81db0a0 con 0x7fe2c80b4d60 2024-03-25T19:41:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.370+0000 7fe2be7fc640 1 -- 172.21.15.69:0/1180581565 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fe2b8052130 con 0x7fe2c80b4d60 2024-03-25T19:41:30.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.370+0000 7fe2be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:30.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.370+0000 7fe2be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:30.371 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:41:30.371 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:30.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.372+0000 7fe2cfce4640 10 librados: watch_flush enter 2024-03-25T19:41:30.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 10 librados: watch_flush exit 2024-03-25T19:41:30.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-25T19:41:30.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 10 client.4240.objecter successfully canceled tick 2024-03-25T19:41:30.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 >> v1:172.21.15.169:6800/3072548259 conn(0x7fe2ac06e0f0 legacy=0x7fe2ac0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:30.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 10 monclient: shutdown 2024-03-25T19:41:30.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:30.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 >> v1:172.21.15.169:6789/0 conn(0x7fe2c80b4d60 legacy=0x7fe2c80a96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:30.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 shutdown_connections 2024-03-25T19:41:30.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 >> 172.21.15.69:0/1180581565 conn(0x7fe2c80617a0 msgr2=0x7fe2c813beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:30.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 shutdown_connections 2024-03-25T19:41:30.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 1 -- 172.21.15.69:0/1180581565 wait complete. 2024-03-25T19:41:30.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:30.373+0000 7fe2cfce4640 1 librados: shutdown 2024-03-25T19:41:30.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:37.191 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.305+0000 7f4ef8493640 10 monclient: get_monmap_and_config 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.305+0000 7f4ef8493640 10 monclient: build_initial_monmap 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.305+0000 7f4ef8493640 10 monclient: monmap: 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:41:37.306436+0000 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:41:37.306436+0000 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:37.305 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:37.306 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:37.306 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:41:37.306 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:41:37.306 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:37.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.305+0000 7f4ef8493640 1 Processor -- start 2024-03-25T19:41:37.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.305+0000 7f4ef8493640 1 -- start start 2024-03-25T19:41:37.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.305+0000 7f4ef8493640 10 monclient: init 2024-03-25T19:41:37.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.306+0000 7f4ef8493640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.306+0000 7f4ef8493640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.306+0000 7f4ef8493640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef00a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.306+0000 7f4ef8493640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef013f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.306+0000 7f4ef8493640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.306+0000 7f4ef8493640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef0142520 con 0x7f4ef013f710 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.307+0000 7f4ef8493640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00b28a0 con 0x7f4ef01430c0 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.307+0000 7f4ef8493640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00b2a30 con 0x7f4ef00a98f0 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.307+0000 7f4ef8493640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:37.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.307+0000 7f4ef8493640 10 monclient(hunting): authenticate will time out at 4909.575906s 2024-03-25T19:41:37.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.313+0000 7f4ef622f640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f4ef00a98f0 0x7f4ef00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:42476/0 (socket says 172.21.15.69:42476) 2024-03-25T19:41:37.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.313+0000 7f4ef622f640 1 -- 172.21.15.69:0/1224474346 learned_addr learned my addr 172.21.15.69:0/1224474346 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:37.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.328+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914646394 0 0) 0x7f4ef00b2a30 con 0x7f4ef00a98f0 2024-03-25T19:41:37.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.328+0000 7f4ef522d640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.328+0000 7f4ef522d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.328+0000 7f4ef522d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.328+0000 7f4ef522d640 10 monclient(hunting): my global_id is 4247 2024-03-25T19:41:37.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.328+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4edc003520 con 0x7f4ef00a98f0 2024-03-25T19:41:37.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.336+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862094090 0 0) 0x7f4ef00b28a0 con 0x7f4ef01430c0 2024-03-25T19:41:37.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.336+0000 7f4ef522d640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.336+0000 7f4ef522d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.336+0000 7f4ef522d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.336+0000 7f4ef522d640 10 monclient(hunting): my global_id is 4243 2024-03-25T19:41:37.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.336+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef00b2a30 con 0x7f4ef01430c0 2024-03-25T19:41:37.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.344+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533377168 0 0) 0x7f4ef0142520 con 0x7f4ef013f710 2024-03-25T19:41:37.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.344+0000 7f4ef522d640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.344+0000 7f4ef522d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.344+0000 7f4ef522d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.344+0000 7f4ef522d640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:41:37.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.344+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef00b28a0 con 0x7f4ef013f710 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1825859546 0 0) 0x7f4ef00b2a30 con 0x7f4ef01430c0 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 >> v1:172.21.15.69:6790/0 conn(0x7f4ef00a98f0 legacy=0x7f4ef00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 >> v1:172.21.15.69:6789/0 conn(0x7f4ef013f710 legacy=0x7f4ef01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 10 monclient: _finish_hunting 0 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 1 monclient: found mon.noname-c 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00b44c0 con 0x7f4ef01430c0 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 10 monclient: _finish_auth 0 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 10 monclient: _check_auth_tickets 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:37.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4ee80019f0 con 0x7f4ef01430c0 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef8493640 5 monclient: authenticate success, global_id 4243 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef8493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef522d640 10 monclient: dump: 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:37.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.345+0000 7f4ef8493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:37.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ee8001bd0 con 0x7f4ef01430c0 2024-03-25T19:41:37.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef522d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef522d640 1 -- 172.21.15.69:0/1224474346 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4ee8001fe0 con 0x7f4ef01430c0 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef522d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef522d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef522d640 10 monclient: dump: 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:37.372 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:37.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef8493640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:37.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef8493640 4 set_mon_vals no callback set 2024-03-25T19:41:37.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef8493640 10 monclient: shutdown 2024-03-25T19:41:37.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef8493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:37.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.372+0000 7f4ef8493640 1 -- 172.21.15.69:0/1224474346 >> v1:172.21.15.169:6789/0 conn(0x7f4ef01430c0 legacy=0x7f4ef014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.373+0000 7f4ef8493640 1 -- 172.21.15.69:0/1224474346 shutdown_connections 2024-03-25T19:41:37.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.373+0000 7f4ef8493640 1 -- 172.21.15.69:0/1224474346 >> 172.21.15.69:0/1224474346 conn(0x7f4ef00b7ea0 msgr2=0x7f4ef00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.373+0000 7f4ef8493640 1 -- 172.21.15.69:0/1224474346 shutdown_connections 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.373+0000 7f4ef8493640 1 -- 172.21.15.69:0/1224474346 wait complete. 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 10 monclient: build_initial_monmap 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 10 monclient: monmap: 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 1 librados: starting msgr at 2024-03-25T19:41:37.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 1 librados: starting objecter 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 1 Processor -- start 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 1 -- start start 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 1 librados: setting wanted keys 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 1 librados: calling monclient init 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.374+0000 7f4ef8493640 10 monclient: init 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef00a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef013f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef01430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00b1820 con 0x7f4ef01430c0 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00ad2e0 con 0x7f4ef013f710 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef01dabb0 con 0x7f4ef00a98f0 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:37.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.375+0000 7f4ef8493640 10 monclient(hunting): authenticate will time out at 4909.644089s 2024-03-25T19:41:37.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.382+0000 7f4ef6a30640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4ef01430c0 0x7f4ef01d7a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35864/0 (socket says 172.21.15.69:35864) 2024-03-25T19:41:37.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.382+0000 7f4ef6a30640 1 -- 172.21.15.69:0/2152188637 learned_addr learned my addr 172.21.15.69:0/2152188637 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:37.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.383+0000 7f4ee6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ef01430c0 2024-03-25T19:41:37.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.383+0000 7f4ee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:37.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.394+0000 7f4ee6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ef00a98f0 2024-03-25T19:41:37.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.394+0000 7f4ee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:37.398 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.398+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826185141 0 0) 0x7f4ef00b1820 con 0x7f4ef01430c0 2024-03-25T19:41:37.398 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.398+0000 7f4ee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.398 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.398+0000 7f4ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.398+0000 7f4ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.398+0000 7f4ee6ffd640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:41:37.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.398+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed4004eb0 con 0x7f4ef01430c0 2024-03-25T19:41:37.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.401+0000 7f4ee6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ef013f710 2024-03-25T19:41:37.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.401+0000 7f4ee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:37.411 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.411+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148533469 0 0) 0x7f4ef00ad2e0 con 0x7f4ef013f710 2024-03-25T19:41:37.411 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.411+0000 7f4ee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.411 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.411+0000 7f4ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.411 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.411+0000 7f4ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.411 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.411+0000 7f4ee6ffd640 10 monclient(hunting): my global_id is 4246 2024-03-25T19:41:37.411 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.411+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef00b1820 con 0x7f4ef013f710 2024-03-25T19:41:37.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.413+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126225513 0 0) 0x7f4ef01dabb0 con 0x7f4ef00a98f0 2024-03-25T19:41:37.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.413+0000 7f4ee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.413+0000 7f4ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.413+0000 7f4ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.413+0000 7f4ee6ffd640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:41:37.414 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.413+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef00ad2e0 con 0x7f4ef00a98f0 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.414+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2288420988 0 0) 0x7f4ed4004eb0 con 0x7f4ef01430c0 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.414+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ef01dabb0 con 0x7f4ef01430c0 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.414+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4ee8001fe0 con 0x7f4ef01430c0 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.414+0000 7f4ee6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.415+0000 7f4ee6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.415+0000 7f4ee6ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:37.415 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:37.416 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:37.416 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:37.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.415+0000 7f4ee6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:37.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.420+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 800101812 0 0) 0x7f4ef00ad2e0 con 0x7f4ef00a98f0 2024-03-25T19:41:37.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.420+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ed4004eb0 con 0x7f4ef00a98f0 2024-03-25T19:41:37.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.420+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4eec001910 con 0x7f4ef00a98f0 2024-03-25T19:41:37.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.420+0000 7f4ee6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.420+0000 7f4ee6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:37.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.420+0000 7f4ee6ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:37.420 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:37.420 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:37.421 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.421 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.421 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.421 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:37.421 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:37.421 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:37.421 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:37.421 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1361635338 0 0) 0x7f4ef01dabb0 con 0x7f4ef01430c0 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 >> v1:172.21.15.69:6790/0 conn(0x7f4ef00a98f0 legacy=0x7f4ef01db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 >> v1:172.21.15.169:6789/0 conn(0x7f4ef013f710 legacy=0x7f4ef01d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 1 monclient: found mon.a 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef01da880 con 0x7f4ef01430c0 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 5 monclient: authenticate success, global_id 4269 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 10 monclient: _renew_subs 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef01db750 con 0x7f4ef01430c0 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 10 monclient: _renew_subs 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef01d82b0 con 0x7f4ef01430c0 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.426+0000 7f4ef8493640 1 librados: init done 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.427+0000 7f4ef8493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.427+0000 7f4ef8493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.427+0000 7f4ef8493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.427+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef014f2d0 con 0x7f4ef01430c0 2024-03-25T19:41:37.734 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.734+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ee8003c20 con 0x7f4ef01430c0 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.734+0000 7f4ee6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.734+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4ee8003f30 con 0x7f4ef01430c0 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.734+0000 7f4ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.734+0000 7f4ee6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.734+0000 7f4ee6ffd640 10 monclient: dump: 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.735 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:37.736 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:37.736 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:37.736 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:37.736 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:37.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.734+0000 7f4ef4a2c640 4 set_mon_vals no callback set 2024-03-25T19:41:37.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.735+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f4ee8004150 con 0x7f4ef01430c0 2024-03-25T19:41:37.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.736+0000 7f4ef622f640 10 monclient: get_auth_request con 0x7f4ed406def0 auth_method 0 2024-03-25T19:41:37.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.736+0000 7f4ee6ffd640 10 client.4269.objecter ms_handle_connect 0x7f4ed406def0 2024-03-25T19:41:37.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.742+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f4ee8054c00 con 0x7f4ef01430c0 2024-03-25T19:41:37.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.742+0000 7f4ee6ffd640 10 client.4269.objecter ms_dispatch 0x7f4ef0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:37.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.742+0000 7f4ee6ffd640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:37.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.742+0000 7f4ee6ffd640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:37.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.742+0000 7f4ee6ffd640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-25T19:41:37.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.742+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ee8050540 con 0x7f4ef01430c0 2024-03-25T19:41:37.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.742+0000 7f4ee6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:37.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.742+0000 7f4ee6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:37.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.978+0000 7f4ef8493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:37.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.978+0000 7f4ef8493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:37.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.978+0000 7f4ef8493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:41:37.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:37.978+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ef00627b0 con 0x7f4ef01430c0 2024-03-25T19:41:38.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.017+0000 7f4ee6ffd640 1 -- 172.21.15.69:0/2152188637 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f4ee8052640 con 0x7f4ef01430c0 2024-03-25T19:41:38.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.017+0000 7f4ee6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:38.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.017+0000 7f4ee6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:38.018 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:41:38.018 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:38.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 10 librados: watch_flush enter 2024-03-25T19:41:38.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 10 librados: watch_flush exit 2024-03-25T19:41:38.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 10 client.4269.objecter successfully canceled tick 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 >> v1:172.21.15.169:6800/3072548259 conn(0x7f4ed406def0 legacy=0x7f4ed40703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 10 monclient: shutdown 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 >> v1:172.21.15.69:6789/0 conn(0x7f4ef01430c0 legacy=0x7f4ef01d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 shutdown_connections 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 >> 172.21.15.69:0/2152188637 conn(0x7f4ef00b7ea0 msgr2=0x7f4ef0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 shutdown_connections 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 1 -- 172.21.15.69:0/2152188637 wait complete. 2024-03-25T19:41:38.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:38.020+0000 7f4ef8493640 1 librados: shutdown 2024-03-25T19:41:38.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:41.338 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:19.302 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:41.372 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:19.336 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:41.404 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:19.368 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:41.440 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:19.394 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:41.477 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:19.441 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:41.502 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:19.456 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:41.646 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:19.611 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:41.719 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:19.683 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:41.724 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:19.679 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:44.838 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.949+0000 7fab30263640 10 monclient: get_monmap_and_config 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.949+0000 7fab30263640 10 monclient: build_initial_monmap 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.949+0000 7fab30263640 10 monclient: monmap: 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:41:44.951106+0000 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:41:44.951106+0000 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:44.950 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:44.951 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:41:44.951 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:41:44.951 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:44.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.951+0000 7fab30263640 1 Processor -- start 2024-03-25T19:41:44.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.951+0000 7fab30263640 1 -- start start 2024-03-25T19:41:44.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.951+0000 7fab30263640 10 monclient: init 2024-03-25T19:41:44.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:44.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:44.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 10 monclient(hunting): picked mon.noname-c con 0x7fab280a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:44.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 10 monclient(hunting): picked mon.noname-b con 0x7fab2813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:44.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 10 monclient(hunting): picked mon.noname-a con 0x7fab281430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:44.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab28142520 con 0x7fab281430c0 2024-03-25T19:41:44.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab280b28a0 con 0x7fab280a98f0 2024-03-25T19:41:44.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab280b2a30 con 0x7fab2813f710 2024-03-25T19:41:44.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:44.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.952+0000 7fab30263640 10 monclient(hunting): authenticate will time out at 4917.221093s 2024-03-25T19:41:44.964 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.965+0000 7fab2e800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fab281430c0 0x7fab2814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53858/0 (socket says 172.21.15.69:53858) 2024-03-25T19:41:44.964 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.965+0000 7fab2e800640 1 -- 172.21.15.69:0/1031139705 learned_addr learned my addr 172.21.15.69:0/1031139705 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:44.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.973+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250258851 0 0) 0x7fab28142520 con 0x7fab281430c0 2024-03-25T19:41:44.974 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.974+0000 7fab2cffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.974 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.974+0000 7fab2cffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.974 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.974+0000 7fab2cffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.974 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.974+0000 7fab2cffd640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:41:44.974 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.974+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab14003520 con 0x7fab281430c0 2024-03-25T19:41:44.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.991+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909984242 0 0) 0x7fab280b2a30 con 0x7fab2813f710 2024-03-25T19:41:44.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.991+0000 7fab2cffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.991+0000 7fab2cffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.991+0000 7fab2cffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.991+0000 7fab2cffd640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:41:44.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:44.991+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab28142520 con 0x7fab2813f710 2024-03-25T19:41:45.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.000+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196773085 0 0) 0x7fab280b28a0 con 0x7fab280a98f0 2024-03-25T19:41:45.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.000+0000 7fab2cffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:45.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.000+0000 7fab2cffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:45.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.000+0000 7fab2cffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:45.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.000+0000 7fab2cffd640 10 monclient(hunting): my global_id is 4249 2024-03-25T19:41:45.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.000+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab280b2a30 con 0x7fab280a98f0 2024-03-25T19:41:45.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3953805343 0 0) 0x7fab28142520 con 0x7fab2813f710 2024-03-25T19:41:45.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 >> v1:172.21.15.169:6789/0 conn(0x7fab280a98f0 legacy=0x7fab280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 >> v1:172.21.15.69:6789/0 conn(0x7fab281430c0 legacy=0x7fab2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:45.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 1 monclient: found mon.noname-b 2024-03-25T19:41:45.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:45.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab280b44c0 con 0x7fab2813f710 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fab18001670 con 0x7fab2813f710 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab2cffd640 10 monclient: dump: 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.003 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:45.004 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:45.004 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:45.004 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:45.004 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:45.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab30263640 5 monclient: authenticate success, global_id 4253 2024-03-25T19:41:45.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.002+0000 7fab30263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.018+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab180017d0 con 0x7fab2813f710 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.018+0000 7fab2cffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.018+0000 7fab2cffd640 1 -- 172.21.15.69:0/1031139705 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fab18001be0 con 0x7fab2813f710 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.018+0000 7fab30263640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.018+0000 7fab30263640 4 set_mon_vals no callback set 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.018+0000 7fab2cffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.018+0000 7fab2cffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.018+0000 7fab2cffd640 10 monclient: dump: 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.019+0000 7fab30263640 10 monclient: shutdown 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.019+0000 7fab30263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.019+0000 7fab30263640 1 -- 172.21.15.69:0/1031139705 >> v1:172.21.15.69:6790/0 conn(0x7fab2813f710 legacy=0x7fab281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.019+0000 7fab30263640 1 -- 172.21.15.69:0/1031139705 shutdown_connections 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.019+0000 7fab30263640 1 -- 172.21.15.69:0/1031139705 >> 172.21.15.69:0/1031139705 conn(0x7fab280b7ea0 msgr2=0x7fab280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.019+0000 7fab30263640 1 -- 172.21.15.69:0/1031139705 shutdown_connections 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.019+0000 7fab30263640 1 -- 172.21.15.69:0/1031139705 wait complete. 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 10 monclient: build_initial_monmap 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 10 monclient: monmap: 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:41:45.022 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 1 librados: starting msgr at 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 1 librados: starting objecter 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 1 Processor -- start 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 1 -- start start 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 1 librados: setting wanted keys 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 1 librados: calling monclient init 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.020+0000 7fab30263640 10 monclient: init 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:45.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 10 monclient(hunting): picked mon.noname-b con 0x7fab280a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 10 monclient(hunting): picked mon.noname-c con 0x7fab2813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 10 monclient(hunting): picked mon.noname-a con 0x7fab281430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab28138ed0 con 0x7fab281430c0 2024-03-25T19:41:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab280abc80 con 0x7fab280a98f0 2024-03-25T19:41:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab280abf60 con 0x7fab2813f710 2024-03-25T19:41:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.021+0000 7fab30263640 10 monclient(hunting): authenticate will time out at 4917.290309s 2024-03-25T19:41:45.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.035+0000 7fab2d7fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fab2813f710 0x7fab280ac660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:41872/0 (socket says 172.21.15.69:41872) 2024-03-25T19:41:45.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.035+0000 7fab2d7fe640 1 -- 172.21.15.69:0/2628399724 learned_addr learned my addr 172.21.15.69:0/2628399724 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:45.038 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.038+0000 7fab1e7fc640 10 client.?.objecter ms_handle_connect 0x7fab2813f710 2024-03-25T19:41:45.038 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.038+0000 7fab1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:45.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.043+0000 7fab1e7fc640 10 client.?.objecter ms_handle_connect 0x7fab280a98f0 2024-03-25T19:41:45.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.043+0000 7fab1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:45.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.049+0000 7fab1e7fc640 10 client.?.objecter ms_handle_connect 0x7fab281430c0 2024-03-25T19:41:45.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.049+0000 7fab1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:45.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.052+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79834238 0 0) 0x7fab280abc80 con 0x7fab280a98f0 2024-03-25T19:41:45.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.052+0000 7fab1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:45.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.052+0000 7fab1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:45.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.052+0000 7fab1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:45.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.052+0000 7fab1e7fc640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:41:45.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.052+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0c004eb0 con 0x7fab280a98f0 2024-03-25T19:41:45.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.054+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355457664 0 0) 0x7fab280abf60 con 0x7fab2813f710 2024-03-25T19:41:45.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.054+0000 7fab1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:45.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.054+0000 7fab1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:45.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.054+0000 7fab1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:45.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.054+0000 7fab1e7fc640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:41:45.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.054+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab280abc80 con 0x7fab2813f710 2024-03-25T19:41:45.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.066+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1016822672 0 0) 0x7fab0c004eb0 con 0x7fab280a98f0 2024-03-25T19:41:45.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.066+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab280abf60 con 0x7fab280a98f0 2024-03-25T19:41:45.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.066+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fab240018f0 con 0x7fab280a98f0 2024-03-25T19:41:45.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.066+0000 7fab1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.066+0000 7fab1e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:45.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.066+0000 7fab1e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:45.067 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.066+0000 7fab1e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:45.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1128416257 0 0) 0x7fab280abc80 con 0x7fab2813f710 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab0c004eb0 con 0x7fab2813f710 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fab18001670 con 0x7fab2813f710 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:45.074 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:45.075 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:45.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310919265 0 0) 0x7fab28138ed0 con 0x7fab281430c0 2024-03-25T19:41:45.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:45.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:45.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:45.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:41:45.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.071+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab280abc80 con 0x7fab281430c0 2024-03-25T19:41:45.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4174226929 0 0) 0x7fab0c004eb0 con 0x7fab2813f710 2024-03-25T19:41:45.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 >> v1:172.21.15.169:6789/0 conn(0x7fab280a98f0 legacy=0x7fab281387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 >> v1:172.21.15.69:6789/0 conn(0x7fab281430c0 legacy=0x7fab281e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 1 monclient: found mon.c 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab28139170 con 0x7fab2813f710 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 5 monclient: authenticate success, global_id 4256 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 10 monclient: _renew_subs 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab280acd70 con 0x7fab2813f710 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 10 monclient: _renew_subs 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:45.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab28136c30 con 0x7fab2813f710 2024-03-25T19:41:45.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.084+0000 7fab30263640 1 librados: init done 2024-03-25T19:41:45.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.085+0000 7fab30263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:45.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.085+0000 7fab30263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:45.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.085+0000 7fab30263640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:45.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.085+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab28136690 con 0x7fab2813f710 2024-03-25T19:41:45.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.396+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab180038a0 con 0x7fab2813f710 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.396+0000 7fab1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.396+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fab18003cb0 con 0x7fab2813f710 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.396+0000 7fab1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.396+0000 7fab1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.396+0000 7fab1e7fc640 10 monclient: dump: 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:45.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.396+0000 7fab1ffff640 4 set_mon_vals no callback set 2024-03-25T19:41:45.398 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.397+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fab18003ed0 con 0x7fab2813f710 2024-03-25T19:41:45.398 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.398+0000 7fab2dfff640 10 monclient: get_auth_request con 0x7fab0c06df30 auth_method 0 2024-03-25T19:41:45.398 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.398+0000 7fab1e7fc640 10 client.4256.objecter ms_handle_connect 0x7fab0c06df30 2024-03-25T19:41:45.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.407+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fab180548b0 con 0x7fab2813f710 2024-03-25T19:41:45.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.407+0000 7fab1e7fc640 10 client.4256.objecter ms_dispatch 0x7fab28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:45.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.407+0000 7fab1e7fc640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:45.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.407+0000 7fab1e7fc640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:45.408 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.407+0000 7fab1e7fc640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-25T19:41:45.408 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.407+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab180503c0 con 0x7fab2813f710 2024-03-25T19:41:45.408 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.407+0000 7fab1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:45.408 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.407+0000 7fab1e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:45.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.681+0000 7fab30263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:45.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.681+0000 7fab30263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:45.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.681+0000 7fab30263640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:41:45.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.681+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab280627b0 con 0x7fab2813f710 2024-03-25T19:41:45.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.715+0000 7fab1e7fc640 1 -- 172.21.15.69:0/2628399724 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fab180522f0 con 0x7fab2813f710 2024-03-25T19:41:45.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.715+0000 7fab1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:45.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.715+0000 7fab1e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:45.716 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:41:45.716 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 10 librados: watch_flush enter 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 10 librados: watch_flush exit 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 10 client.4256.objecter successfully canceled tick 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 >> v1:172.21.15.169:6800/3072548259 conn(0x7fab0c06df30 legacy=0x7fab0c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 10 monclient: shutdown 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 >> v1:172.21.15.69:6790/0 conn(0x7fab2813f710 legacy=0x7fab280ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 shutdown_connections 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.718+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 >> 172.21.15.69:0/2628399724 conn(0x7fab280b7ea0 msgr2=0x7fab28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.719+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 shutdown_connections 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.719+0000 7fab30263640 1 -- 172.21.15.69:0/2628399724 wait complete. 2024-03-25T19:41:45.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:45.719+0000 7fab30263640 1 librados: shutdown 2024-03-25T19:41:45.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:52.538 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:52.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.639+0000 7f183003e640 10 monclient: get_monmap_and_config 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.639+0000 7f183003e640 10 monclient: build_initial_monmap 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.639+0000 7f183003e640 10 monclient: monmap: 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:41:52.640943+0000 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:41:52.640943+0000 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:52.640 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:52.641 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:41:52.641 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:41:52.641 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:52.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.640+0000 7f183003e640 1 Processor -- start 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.640+0000 7f183003e640 1 -- start start 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.640+0000 7f183003e640 10 monclient: init 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 10 monclient(hunting): picked mon.noname-b con 0x7f18280a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 10 monclient(hunting): picked mon.noname-c con 0x7f182813f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 10 monclient(hunting): picked mon.noname-a con 0x7f18281430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828142520 con 0x7f18281430c0 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280b28a0 con 0x7f182813f710 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280b2a30 con 0x7f18280a98f0 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.641+0000 7f183003e640 10 monclient(hunting): authenticate will time out at 4924.910452s 2024-03-25T19:41:52.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.645+0000 7f182ddda640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f18280a98f0 0x7f18280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:43150/0 (socket says 172.21.15.69:43150) 2024-03-25T19:41:52.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.645+0000 7f182ddda640 1 -- 172.21.15.69:0/2419946489 learned_addr learned my addr 172.21.15.69:0/2419946489 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:52.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.654+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064740177 0 0) 0x7f18280b2a30 con 0x7f18280a98f0 2024-03-25T19:41:52.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.654+0000 7f182cdd8640 10 monclient(hunting): _negotiate 2024-03-25T19:41:52.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.654+0000 7f182cdd8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:52.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.654+0000 7f182cdd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:52.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.654+0000 7f182cdd8640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:41:52.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.654+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1814003520 con 0x7f18280a98f0 2024-03-25T19:41:52.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142583464 0 0) 0x7f1814003520 con 0x7f18280a98f0 2024-03-25T19:41:52.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 >> v1:172.21.15.169:6789/0 conn(0x7f182813f710 legacy=0x7f18281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 >> v1:172.21.15.69:6789/0 conn(0x7f18281430c0 legacy=0x7f182814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 10 monclient: _finish_hunting 0 2024-03-25T19:41:52.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 1 monclient: found mon.noname-b 2024-03-25T19:41:52.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18280b44c0 con 0x7f18280a98f0 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 10 monclient: _finish_auth 0 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 10 monclient: _check_auth_tickets 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1824001690 con 0x7f18280a98f0 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f183003e640 5 monclient: authenticate success, global_id 4259 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f183003e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f182cdd8640 10 monclient: dump: 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:52.669 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:52.670 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:52.670 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.670 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:52.670 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:52.670 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:52.670 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:52.670 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:52.670 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.668+0000 7f183003e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.673+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18240017f0 con 0x7f18280a98f0 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.673+0000 7f182cdd8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.673+0000 7f182cdd8640 1 -- 172.21.15.69:0/2419946489 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1824001c00 con 0x7f18280a98f0 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.673+0000 7f183003e640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.673+0000 7f183003e640 4 set_mon_vals no callback set 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.673+0000 7f182cdd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.673+0000 7f182cdd8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.673+0000 7f182cdd8640 10 monclient: dump: 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:52.673 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:52.674 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:52.674 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:52.674 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.674 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:52.674 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:52.674 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:52.674 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:52.674 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:52.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 10 monclient: shutdown 2024-03-25T19:41:52.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:52.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 1 -- 172.21.15.69:0/2419946489 >> v1:172.21.15.69:6790/0 conn(0x7f18280a98f0 legacy=0x7f18280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 1 -- 172.21.15.69:0/2419946489 shutdown_connections 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 1 -- 172.21.15.69:0/2419946489 >> 172.21.15.69:0/2419946489 conn(0x7f18280b7ea0 msgr2=0x7f18280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 1 -- 172.21.15.69:0/2419946489 shutdown_connections 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 1 -- 172.21.15.69:0/2419946489 wait complete. 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 10 monclient: build_initial_monmap 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.674+0000 7f183003e640 10 monclient: monmap: 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:41:52.677 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.675+0000 7f183003e640 1 librados: starting msgr at 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.675+0000 7f183003e640 1 librados: starting objecter 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.675+0000 7f183003e640 1 Processor -- start 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.675+0000 7f183003e640 1 -- start start 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.675+0000 7f183003e640 1 librados: setting wanted keys 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.675+0000 7f183003e640 1 librados: calling monclient init 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.675+0000 7f183003e640 10 monclient: init 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 10 monclient(hunting): picked mon.noname-c con 0x7f18280a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 10 monclient(hunting): picked mon.noname-a con 0x7f182813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 10 monclient(hunting): picked mon.noname-b con 0x7f18281430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:41:52.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828139a40 con 0x7f182813f710 2024-03-25T19:41:52.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828133f20 con 0x7f18281430c0 2024-03-25T19:41:52.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280ab950 con 0x7f18280a98f0 2024-03-25T19:41:52.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:52.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.676+0000 7f183003e640 10 monclient(hunting): authenticate will time out at 4924.944819s 2024-03-25T19:41:52.682 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.682+0000 7f182ddda640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f18280a98f0 0x7f1828138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:43172/0 (socket says 172.21.15.69:43172) 2024-03-25T19:41:52.682 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.682+0000 7f182ddda640 1 -- 172.21.15.69:0/2108726946 learned_addr learned my addr 172.21.15.69:0/2108726946 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:41:52.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.685+0000 7f181e7fc640 10 client.?.objecter ms_handle_connect 0x7f18281430c0 2024-03-25T19:41:52.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.685+0000 7f181e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:52.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.687+0000 7f181e7fc640 10 client.?.objecter ms_handle_connect 0x7f18280a98f0 2024-03-25T19:41:52.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.687+0000 7f181e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:52.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.687+0000 7f181e7fc640 10 client.?.objecter ms_handle_connect 0x7f182813f710 2024-03-25T19:41:52.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.687+0000 7f181e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:52.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.690+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996752814 0 0) 0x7f1828133f20 con 0x7f18281430c0 2024-03-25T19:41:52.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.690+0000 7f181e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:52.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.690+0000 7f181e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:52.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.690+0000 7f181e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:52.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.690+0000 7f181e7fc640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:41:52.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.690+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f180c004eb0 con 0x7f18281430c0 2024-03-25T19:41:52.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.697+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394487877 0 0) 0x7f18280ab950 con 0x7f18280a98f0 2024-03-25T19:41:52.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.697+0000 7f181e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:52.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.697+0000 7f181e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:52.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.697+0000 7f181e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:52.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.697+0000 7f181e7fc640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:41:52.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.697+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1828133f20 con 0x7f18280a98f0 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.708+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4056804338 0 0) 0x7f180c004eb0 con 0x7f18281430c0 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.708+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18280ab950 con 0x7f18281430c0 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.708+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1820001b90 con 0x7f18281430c0 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.708+0000 7f181e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.708+0000 7f181e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.708+0000 7f181e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:52.708 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.709 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:52.709 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:52.709 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:52.709 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:52.709 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:52.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.708+0000 7f181e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:52.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.709+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966780721 0 0) 0x7f1828139a40 con 0x7f182813f710 2024-03-25T19:41:52.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.709+0000 7f181e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:52.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.709+0000 7f181e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:52.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.709+0000 7f181e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:52.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.709+0000 7f181e7fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:41:52.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.709+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f180c004eb0 con 0x7f182813f710 2024-03-25T19:41:52.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.715+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3964616364 0 0) 0x7f1828133f20 con 0x7f18280a98f0 2024-03-25T19:41:52.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.715+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1828139a40 con 0x7f18280a98f0 2024-03-25T19:41:52.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.715+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1824001b40 con 0x7f18280a98f0 2024-03-25T19:41:52.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.715+0000 7f181e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.715+0000 7f181e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:52.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.715+0000 7f181e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:52.715 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:52.716 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 56911123 0 0) 0x7f18280ab950 con 0x7f18281430c0 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 >> v1:172.21.15.69:6790/0 conn(0x7f18280a98f0 legacy=0x7f1828138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 >> v1:172.21.15.69:6789/0 conn(0x7f182813f710 legacy=0x7f1828133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 1 monclient: found mon.b 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18280abc30 con 0x7f18281430c0 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f181e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:52.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 5 monclient: authenticate success, global_id 4258 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 10 monclient: _renew_subs 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18280abf10 con 0x7f18281430c0 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 10 monclient: _renew_subs 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1828139370 con 0x7f18281430c0 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.716+0000 7f183003e640 1 librados: init done 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.717+0000 7f183003e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.717+0000 7f183003e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.717+0000 7f183003e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:52.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:52.717+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18280627b0 con 0x7f18281430c0 2024-03-25T19:41:53.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.011+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1820002080 con 0x7f18281430c0 2024-03-25T19:41:53.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.011+0000 7f181e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:53.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.011+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1820003b10 con 0x7f18281430c0 2024-03-25T19:41:53.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.011+0000 7f181e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:53.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.011+0000 7f181e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:53.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.011+0000 7f181e7fc640 10 monclient: dump: 2024-03-25T19:41:53.011 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:41:53.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.011+0000 7f181ffff640 4 set_mon_vals no callback set 2024-03-25T19:41:53.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.012+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f1820003d30 con 0x7f18281430c0 2024-03-25T19:41:53.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.012+0000 7f182ddda640 10 monclient: get_auth_request con 0x7f180c06dfb0 auth_method 0 2024-03-25T19:41:53.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.012+0000 7f181e7fc640 10 client.4258.objecter ms_handle_connect 0x7f180c06dfb0 2024-03-25T19:41:53.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.018+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f1820053870 con 0x7f18281430c0 2024-03-25T19:41:53.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.018+0000 7f181e7fc640 10 client.4258.objecter ms_dispatch 0x7f1828000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:53.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.018+0000 7f181e7fc640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:53.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.018+0000 7f181e7fc640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:53.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.018+0000 7f181e7fc640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T19:41:53.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.019+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f182004f380 con 0x7f18281430c0 2024-03-25T19:41:53.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.019+0000 7f181e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:53.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.019+0000 7f181e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:53.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.265+0000 7f183003e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:53.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.265+0000 7f183003e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:53.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.265+0000 7f183003e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:41:53.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.265+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18280b7180 con 0x7f18281430c0 2024-03-25T19:41:53.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.311+0000 7f181e7fc640 1 -- 172.21.15.69:0/2108726946 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f18200512b0 con 0x7f18281430c0 2024-03-25T19:41:53.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.311+0000 7f181e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:53.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.311+0000 7f181e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:53.311 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:41:53.311 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:53.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.313+0000 7f183003e640 10 librados: watch_flush enter 2024-03-25T19:41:53.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.313+0000 7f183003e640 10 librados: watch_flush exit 2024-03-25T19:41:53.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.313+0000 7f183003e640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T19:41:53.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.313+0000 7f183003e640 10 client.4258.objecter successfully canceled tick 2024-03-25T19:41:53.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 >> v1:172.21.15.169:6800/3072548259 conn(0x7f180c06dfb0 legacy=0x7f180c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:53.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 10 monclient: shutdown 2024-03-25T19:41:53.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:53.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 >> v1:172.21.15.169:6789/0 conn(0x7f18281430c0 legacy=0x7f1828133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:53.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 shutdown_connections 2024-03-25T19:41:53.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 >> 172.21.15.69:0/2108726946 conn(0x7f18280b7ea0 msgr2=0x7f1828074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:53.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 shutdown_connections 2024-03-25T19:41:53.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 1 -- 172.21.15.69:0/2108726946 wait complete. 2024-03-25T19:41:53.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:41:53.314+0000 7f183003e640 1 librados: shutdown 2024-03-25T19:41:53.327 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:00.134 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:00.246 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.247+0000 7f629e3f4640 10 monclient: get_monmap_and_config 2024-03-25T19:42:00.246 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.247+0000 7f629e3f4640 10 monclient: build_initial_monmap 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.247+0000 7f629e3f4640 10 monclient: monmap: 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:42:00.247873+0000 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:42:00.247873+0000 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:42:00.247 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:00.248 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.248+0000 7f629e3f4640 1 Processor -- start 2024-03-25T19:42:00.248 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 1 -- start start 2024-03-25T19:42:00.248 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 10 monclient: init 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 10 monclient(hunting): picked mon.noname-a con 0x7f62980a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 10 monclient(hunting): picked mon.noname-b con 0x7f629813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 10 monclient(hunting): picked mon.noname-c con 0x7f62981430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6298142520 con 0x7f62980a98f0 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62980b28a0 con 0x7f62981430c0 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62980b2a30 con 0x7f629813f710 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:00.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.249+0000 7f629e3f4640 10 monclient(hunting): authenticate will time out at 4932.517545s 2024-03-25T19:42:00.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.259+0000 7f628ffff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f629813f710 0x7f62981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:47916/0 (socket says 172.21.15.69:47916) 2024-03-25T19:42:00.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.259+0000 7f628ffff640 1 -- 172.21.15.69:0/132351097 learned_addr learned my addr 172.21.15.69:0/132351097 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:00.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.266+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748365958 0 0) 0x7f62980b2a30 con 0x7f629813f710 2024-03-25T19:42:00.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.266+0000 7f62977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.266+0000 7f62977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.266+0000 7f62977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.266+0000 7f62977fe640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:42:00.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.266+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6278003520 con 0x7f629813f710 2024-03-25T19:42:00.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.279+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342209792 0 0) 0x7f6298142520 con 0x7f62980a98f0 2024-03-25T19:42:00.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.279+0000 7f62977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.279+0000 7f62977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.279+0000 7f62977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.279+0000 7f62977fe640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:42:00.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.279+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62980b2a30 con 0x7f62980a98f0 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.284+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 513819290 0 0) 0x7f6278003520 con 0x7f629813f710 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.284+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 >> v1:172.21.15.169:6789/0 conn(0x7f62981430c0 legacy=0x7f629814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.284+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 >> v1:172.21.15.69:6789/0 conn(0x7f62980a98f0 legacy=0x7f62980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.284+0000 7f62977fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.284+0000 7f62977fe640 1 monclient: found mon.noname-b 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.284+0000 7f62977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62980b44c0 con 0x7f629813f710 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:00.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6280001690 con 0x7f629813f710 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f629e3f4640 5 monclient: authenticate success, global_id 4265 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f629e3f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f62977fe640 10 monclient: dump: 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:00.285 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:00.286 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:00.286 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.285+0000 7f629e3f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:00.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.290+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62800017f0 con 0x7f629813f710 2024-03-25T19:42:00.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.290+0000 7f62977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:00.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.290+0000 7f62977fe640 1 -- 172.21.15.69:0/132351097 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6280001c00 con 0x7f629813f710 2024-03-25T19:42:00.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.290+0000 7f629e3f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.290+0000 7f629e3f4640 4 set_mon_vals no callback set 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.290+0000 7f62977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.290+0000 7f62977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.290+0000 7f62977fe640 10 monclient: dump: 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:00.290 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:00.291 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:00.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.291+0000 7f629e3f4640 10 monclient: shutdown 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.291+0000 7f629e3f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.291+0000 7f629e3f4640 1 -- 172.21.15.69:0/132351097 >> v1:172.21.15.69:6790/0 conn(0x7f629813f710 legacy=0x7f62981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.291+0000 7f629e3f4640 1 -- 172.21.15.69:0/132351097 shutdown_connections 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.291+0000 7f629e3f4640 1 -- 172.21.15.69:0/132351097 >> 172.21.15.69:0/132351097 conn(0x7f62980b7ea0 msgr2=0x7f62980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.291+0000 7f629e3f4640 1 -- 172.21.15.69:0/132351097 shutdown_connections 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.291+0000 7f629e3f4640 1 -- 172.21.15.69:0/132351097 wait complete. 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 10 monclient: build_initial_monmap 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 10 monclient: monmap: 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:42:00.293 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 1 librados: starting msgr at 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 1 librados: starting objecter 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 1 Processor -- start 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 1 -- start start 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 1 librados: setting wanted keys 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 1 librados: calling monclient init 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.292+0000 7f629e3f4640 10 monclient: init 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:00.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 10 monclient(hunting): picked mon.noname-b con 0x7f62980a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:00.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 10 monclient(hunting): picked mon.noname-a con 0x7f629813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:00.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 10 monclient(hunting): picked mon.noname-c con 0x7f62981430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:00.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62980aba00 con 0x7f629813f710 2024-03-25T19:42:00.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62981d7490 con 0x7f62980a98f0 2024-03-25T19:42:00.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62981d8360 con 0x7f62981430c0 2024-03-25T19:42:00.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:00.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.293+0000 7f629e3f4640 10 monclient(hunting): authenticate will time out at 4932.561161s 2024-03-25T19:42:00.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.305+0000 7f6297fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f62980a98f0 0x7f62981386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:32870/0 (socket says 172.21.15.69:32870) 2024-03-25T19:42:00.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.305+0000 7f6297fff640 1 -- 172.21.15.69:0/1531320308 learned_addr learned my addr 172.21.15.69:0/1531320308 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:00.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.306+0000 7f62957fa640 10 client.?.objecter ms_handle_connect 0x7f62980a98f0 2024-03-25T19:42:00.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.306+0000 7f62957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.309+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275755574 0 0) 0x7f62981d7490 con 0x7f62980a98f0 2024-03-25T19:42:00.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.309+0000 7f62957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.309+0000 7f62957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.309+0000 7f62957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.309+0000 7f62957fa640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:42:00.308 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.309+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f627c004eb0 con 0x7f62980a98f0 2024-03-25T19:42:00.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.314+0000 7f62957fa640 10 client.?.objecter ms_handle_connect 0x7f62981430c0 2024-03-25T19:42:00.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.314+0000 7f62957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.316+0000 7f62957fa640 10 client.?.objecter ms_handle_connect 0x7f629813f710 2024-03-25T19:42:00.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.316+0000 7f62957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.326+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101104254 0 0) 0x7f62980aba00 con 0x7f629813f710 2024-03-25T19:42:00.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.326+0000 7f62957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.326+0000 7f62957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.326+0000 7f62957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.326+0000 7f62957fa640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:42:00.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.326+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62981d7490 con 0x7f629813f710 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.331+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3696994329 0 0) 0x7f627c004eb0 con 0x7f62980a98f0 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.331+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62980aba00 con 0x7f62980a98f0 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.331+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6288001a10 con 0x7f62980a98f0 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.331+0000 7f62957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.331+0000 7f62957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.331+0000 7f62957fa640 10 monclient(hunting): dump: 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.330 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:00.331 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:00.331 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:00.331 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:00.331 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:00.331 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.331+0000 7f62957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:00.332 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.333+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193235836 0 0) 0x7f62981d8360 con 0x7f62981430c0 2024-03-25T19:42:00.332 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.333+0000 7f62957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.332 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.333+0000 7f62957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.332 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.333+0000 7f62957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.333+0000 7f62957fa640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:42:00.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.333+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f627c004eb0 con 0x7f62981430c0 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.343+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 761634638 0 0) 0x7f62981d7490 con 0x7f629813f710 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.343+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62981d8360 con 0x7f629813f710 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.343+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6280001b00 con 0x7f629813f710 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.343+0000 7f62957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.343+0000 7f62957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.343+0000 7f62957fa640 10 monclient(hunting): dump: 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:00.343 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:00.344 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:00.344 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:00.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.345+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893883348 0 0) 0x7f627c004eb0 con 0x7f62981430c0 2024-03-25T19:42:00.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.345+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62981d7490 con 0x7f62981430c0 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.345+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6290001ad0 con 0x7f62981430c0 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.345+0000 7f62957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.345+0000 7f62957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.345+0000 7f62957fa640 10 monclient(hunting): dump: 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:00.345 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:00.346 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:00.352 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1559116603 0 0) 0x7f62980aba00 con 0x7f62980a98f0 2024-03-25T19:42:00.352 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 >> v1:172.21.15.69:6790/0 conn(0x7f62981430c0 legacy=0x7f6298133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.352 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 >> v1:172.21.15.69:6789/0 conn(0x7f629813f710 legacy=0x7f62981d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 1 monclient: found mon.b 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62980abc30 con 0x7f62980a98f0 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f62957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 5 monclient: authenticate success, global_id 4264 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 10 monclient: _renew_subs 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6298137940 con 0x7f62980a98f0 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 10 monclient: _renew_subs 2024-03-25T19:42:00.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:00.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6298135220 con 0x7f62980a98f0 2024-03-25T19:42:00.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.353+0000 7f629e3f4640 1 librados: init done 2024-03-25T19:42:00.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.355+0000 7f629e3f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.355+0000 7f629e3f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.355+0000 7f629e3f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:00.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.355+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62980627b0 con 0x7f62980a98f0 2024-03-25T19:42:00.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.575+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6288001c50 con 0x7f62980a98f0 2024-03-25T19:42:00.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.575+0000 7f62957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:00.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.575+0000 7f6296ffd640 4 set_mon_vals no callback set 2024-03-25T19:42:00.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.637+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6288003a60 con 0x7f62980a98f0 2024-03-25T19:42:00.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.637+0000 7f62957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.637+0000 7f62957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:00.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.637+0000 7f62957fa640 10 monclient: dump: 2024-03-25T19:42:00.636 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:00.636 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:00.636 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.636 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:00.637 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.637 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:00.637 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:00.637 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:00.637 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:00.637 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:00.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.638+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f6288003c40 con 0x7f62980a98f0 2024-03-25T19:42:00.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.638+0000 7f628ffff640 10 monclient: get_auth_request con 0x7f627c06e160 auth_method 0 2024-03-25T19:42:00.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.639+0000 7f62957fa640 10 client.4264.objecter ms_handle_connect 0x7f627c06e160 2024-03-25T19:42:00.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.640+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f62880548a0 con 0x7f62980a98f0 2024-03-25T19:42:00.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.640+0000 7f62957fa640 10 client.4264.objecter ms_dispatch 0x7f6298000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:00.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.640+0000 7f62957fa640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:00.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.640+0000 7f62957fa640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:00.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.640+0000 7f62957fa640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T19:42:00.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.643+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62880503b0 con 0x7f62980a98f0 2024-03-25T19:42:00.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.643+0000 7f62957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.643+0000 7f62957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:00.863 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.864+0000 7f629e3f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:00.863 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.864+0000 7f629e3f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:00.863 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.864+0000 7f629e3f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:00.863 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.864+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62981d7830 con 0x7f62980a98f0 2024-03-25T19:42:00.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.891+0000 7f62957fa640 1 -- 172.21.15.69:0/1531320308 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f62880522e0 con 0x7f62980a98f0 2024-03-25T19:42:00.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.891+0000 7f62957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:00.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.891+0000 7f62957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:00.890 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:42:00.890 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 10 librados: watch_flush enter 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 10 librados: watch_flush exit 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 10 client.4264.objecter successfully canceled tick 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 >> v1:172.21.15.169:6800/3072548259 conn(0x7f627c06e160 legacy=0x7f627c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 10 monclient: shutdown 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 >> v1:172.21.15.169:6789/0 conn(0x7f62980a98f0 legacy=0x7f62981386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 shutdown_connections 2024-03-25T19:42:00.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.893+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 >> 172.21.15.69:0/1531320308 conn(0x7f62980b7ea0 msgr2=0x7f6298074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:00.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.894+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 shutdown_connections 2024-03-25T19:42:00.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.894+0000 7f629e3f4640 1 -- 172.21.15.69:0/1531320308 wait complete. 2024-03-25T19:42:00.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:00.894+0000 7f629e3f4640 1 librados: shutdown 2024-03-25T19:42:00.903 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:01.318 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:39.282 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:01.366 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:39.330 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:01.417 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:39.381 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:01.440 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:39.405 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:01.445 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:39.409 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:01.446 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:39.401 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:01.454 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:39.409 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:01.504 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:39.458 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:01.508 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:39.472 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:07.710 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:07.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.822+0000 7f70b8f1c640 10 monclient: get_monmap_and_config 2024-03-25T19:42:07.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.822+0000 7f70b8f1c640 10 monclient: build_initial_monmap 2024-03-25T19:42:07.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.822+0000 7f70b8f1c640 10 monclient: monmap: 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:42:07.823949+0000 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:42:07.823949+0000 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:42:07.823 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:07.824 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.823+0000 7f70b8f1c640 1 Processor -- start 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.823+0000 7f70b8f1c640 1 -- start start 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.823+0000 7f70b8f1c640 10 monclient: init 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 10 monclient(hunting): picked mon.noname-b con 0x7f70b406fb30 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 10 monclient(hunting): picked mon.noname-a con 0x7f70b413c0b0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 10 monclient(hunting): picked mon.noname-c con 0x7f70b406ff10 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b4071450 con 0x7f70b413c0b0 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b4071730 con 0x7f70b406ff10 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40730f0 con 0x7f70b406fb30 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:07.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.824+0000 7f70b8f1c640 10 monclient(hunting): authenticate will time out at 4940.093384s 2024-03-25T19:42:07.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.840+0000 7f70abfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f70b413c0b0 0x7f70b4071b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51392/0 (socket says 172.21.15.69:51392) 2024-03-25T19:42:07.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.840+0000 7f70abfff640 1 -- 172.21.15.69:0/4194495041 learned_addr learned my addr 172.21.15.69:0/4194495041 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:07.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.852+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940047448 0 0) 0x7f70b4071730 con 0x7f70b406ff10 2024-03-25T19:42:07.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.852+0000 7f70b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.852+0000 7f70b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.852+0000 7f70b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.852+0000 7f70b1d9b640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:42:07.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.852+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a0004630 con 0x7f70b406ff10 2024-03-25T19:42:07.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.862+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271112625 0 0) 0x7f70b40730f0 con 0x7f70b406fb30 2024-03-25T19:42:07.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.862+0000 7f70b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.862+0000 7f70b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.863 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.862+0000 7f70b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.863 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.862+0000 7f70b1d9b640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:42:07.863 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.862+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b4071730 con 0x7f70b406fb30 2024-03-25T19:42:07.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.865+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369183528 0 0) 0x7f70b4071450 con 0x7f70b413c0b0 2024-03-25T19:42:07.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.865+0000 7f70b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.866 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.865+0000 7f70b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.866 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.865+0000 7f70b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.866 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.865+0000 7f70b1d9b640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:42:07.866 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.865+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b40730f0 con 0x7f70b413c0b0 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.868+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 696119008 0 0) 0x7f70a0004630 con 0x7f70b406ff10 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.868+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 >> v1:172.21.15.69:6790/0 conn(0x7f70b406fb30 legacy=0x7f70b413b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 >> v1:172.21.15.69:6789/0 conn(0x7f70b413c0b0 legacy=0x7f70b4071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 1 monclient: found mon.noname-c 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b406ef90 con 0x7f70b406ff10 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:07.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f70ac001a10 con 0x7f70b406ff10 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b1d9b640 10 monclient: dump: 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:07.870 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:07.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b8f1c640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:42:07.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.869+0000 7f70b8f1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:07.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.884+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70ac001b70 con 0x7f70b406ff10 2024-03-25T19:42:07.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.884+0000 7f70b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:07.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.884+0000 7f70b1d9b640 1 -- 172.21.15.69:0/4194495041 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f70ac001f80 con 0x7f70b406ff10 2024-03-25T19:42:07.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.884+0000 7f70b8f1c640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.884+0000 7f70b8f1c640 4 set_mon_vals no callback set 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.884+0000 7f70b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.884+0000 7f70b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.884+0000 7f70b1d9b640 10 monclient: dump: 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:07.885 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.885+0000 7f70b8f1c640 10 monclient: shutdown 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.885+0000 7f70b8f1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.885+0000 7f70b8f1c640 1 -- 172.21.15.69:0/4194495041 >> v1:172.21.15.169:6789/0 conn(0x7f70b406ff10 legacy=0x7f70b414a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.885+0000 7f70b8f1c640 1 -- 172.21.15.69:0/4194495041 shutdown_connections 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.885+0000 7f70b8f1c640 1 -- 172.21.15.69:0/4194495041 >> 172.21.15.69:0/4194495041 conn(0x7f70b40b11e0 msgr2=0x7f70b40b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.885+0000 7f70b8f1c640 1 -- 172.21.15.69:0/4194495041 shutdown_connections 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.885+0000 7f70b8f1c640 1 -- 172.21.15.69:0/4194495041 wait complete. 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 10 monclient: build_initial_monmap 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 10 monclient: monmap: 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:42:07.888 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 1 librados: starting msgr at 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 1 librados: starting objecter 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 1 Processor -- start 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 1 -- start start 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 1 librados: setting wanted keys 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 1 librados: calling monclient init 2024-03-25T19:42:07.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.886+0000 7f70b8f1c640 10 monclient: init 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 10 monclient(hunting): picked mon.noname-a con 0x7f70b406fb30 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 10 monclient(hunting): picked mon.noname-c con 0x7f70b406ff10 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 10 monclient(hunting): picked mon.noname-b con 0x7f70b413c0b0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b413f620 con 0x7f70b406fb30 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b413e3e0 con 0x7f70b413c0b0 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b413f100 con 0x7f70b406ff10 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:07.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.887+0000 7f70b8f1c640 10 monclient(hunting): authenticate will time out at 4940.156178s 2024-03-25T19:42:07.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.909+0000 7f70b259c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f70b406fb30 0x7f70b4138540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51440/0 (socket says 172.21.15.69:51440) 2024-03-25T19:42:07.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.909+0000 7f70b259c640 1 -- 172.21.15.69:0/3984508874 learned_addr learned my addr 172.21.15.69:0/3984508874 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:07.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.911+0000 7f70aaffd640 10 client.?.objecter ms_handle_connect 0x7f70b413c0b0 2024-03-25T19:42:07.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.911+0000 7f70aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.913+0000 7f70aaffd640 10 client.?.objecter ms_handle_connect 0x7f70b406fb30 2024-03-25T19:42:07.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.913+0000 7f70aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.916+0000 7f70aaffd640 10 client.?.objecter ms_handle_connect 0x7f70b406ff10 2024-03-25T19:42:07.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.916+0000 7f70aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.916+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508232705 0 0) 0x7f70b413e3e0 con 0x7f70b413c0b0 2024-03-25T19:42:07.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.916+0000 7f70aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.916+0000 7f70aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.916+0000 7f70aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.916+0000 7f70aaffd640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:42:07.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.916+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7094004eb0 con 0x7f70b413c0b0 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.924+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2710099923 0 0) 0x7f7094004eb0 con 0x7f70b413c0b0 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.924+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b413e3e0 con 0x7f70b413c0b0 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.924+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f70ac001e80 con 0x7f70b413c0b0 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.924+0000 7f70aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.924+0000 7f70aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.924+0000 7f70aaffd640 10 monclient(hunting): dump: 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:07.924 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.924+0000 7f70aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.925+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273797018 0 0) 0x7f70b413f100 con 0x7f70b406ff10 2024-03-25T19:42:07.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.925+0000 7f70aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.925+0000 7f70aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.925+0000 7f70aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.925+0000 7f70aaffd640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:42:07.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.925+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7094004eb0 con 0x7f70b406ff10 2024-03-25T19:42:07.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.931+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147558767 0 0) 0x7f70b413f620 con 0x7f70b406fb30 2024-03-25T19:42:07.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.931+0000 7f70aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.931+0000 7f70aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.931+0000 7f70aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.931+0000 7f70aaffd640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:42:07.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.931+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b413f100 con 0x7f70b406fb30 2024-03-25T19:42:07.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2347822832 0 0) 0x7f70b413e3e0 con 0x7f70b413c0b0 2024-03-25T19:42:07.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 >> v1:172.21.15.69:6790/0 conn(0x7f70b406ff10 legacy=0x7f70b41d53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 >> v1:172.21.15.69:6789/0 conn(0x7f70b406fb30 legacy=0x7f70b4138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:07.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 1 monclient: found mon.b 2024-03-25T19:42:07.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:07.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:07.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b413f7d0 con 0x7f70b413c0b0 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 10 monclient: _renew_subs 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b413ebe0 con 0x7f70b413c0b0 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 10 monclient: _renew_subs 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b4138e30 con 0x7f70b413c0b0 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.942+0000 7f70b8f1c640 1 librados: init done 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.943+0000 7f70b8f1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:07.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.943+0000 7f70b8f1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:07.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.943+0000 7f70b8f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:07.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:07.943+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b413fa20 con 0x7f70b413c0b0 2024-03-25T19:42:08.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.243+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70ac003c20 con 0x7f70b413c0b0 2024-03-25T19:42:08.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.243+0000 7f70aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:08.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.243+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f70ac003f30 con 0x7f70b413c0b0 2024-03-25T19:42:08.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.243+0000 7f70aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:08.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.243+0000 7f70aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:08.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.243+0000 7f70aaffd640 10 monclient: dump: 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:08.244 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.243+0000 7f70b159a640 4 set_mon_vals no callback set 2024-03-25T19:42:08.245 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.244+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f70ac019f00 con 0x7f70b413c0b0 2024-03-25T19:42:08.245 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.245+0000 7f70b259c640 10 monclient: get_auth_request con 0x7f709406de80 auth_method 0 2024-03-25T19:42:08.245 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.245+0000 7f70aaffd640 10 client.4270.objecter ms_handle_connect 0x7f709406de80 2024-03-25T19:42:08.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.251+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f70ac053a00 con 0x7f70b413c0b0 2024-03-25T19:42:08.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.251+0000 7f70aaffd640 10 client.4270.objecter ms_dispatch 0x7f70b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:08.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.251+0000 7f70aaffd640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:08.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.251+0000 7f70aaffd640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:08.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.251+0000 7f70aaffd640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T19:42:08.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.251+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70ac04f510 con 0x7f70b413c0b0 2024-03-25T19:42:08.252 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.251+0000 7f70aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:08.252 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.251+0000 7f70aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:08.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.496+0000 7f70b8f1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:08.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.496+0000 7f70b8f1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:08.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.496+0000 7f70b8f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:08.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.496+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70b413f290 con 0x7f70b413c0b0 2024-03-25T19:42:08.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.527+0000 7f70aaffd640 1 -- 172.21.15.69:0/3984508874 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f70ac051440 con 0x7f70b413c0b0 2024-03-25T19:42:08.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.528+0000 7f70aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:08.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.528+0000 7f70aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:08.528 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:42:08.528 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:08.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 10 librados: watch_flush enter 2024-03-25T19:42:08.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 10 librados: watch_flush exit 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 10 client.4270.objecter successfully canceled tick 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 >> v1:172.21.15.169:6800/3072548259 conn(0x7f709406de80 legacy=0x7f7094070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 10 monclient: shutdown 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 >> v1:172.21.15.169:6789/0 conn(0x7f70b413c0b0 legacy=0x7f70b41e0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 shutdown_connections 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 >> 172.21.15.69:0/3984508874 conn(0x7f70b40b11e0 msgr2=0x7f70b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 shutdown_connections 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 1 -- 172.21.15.69:0/3984508874 wait complete. 2024-03-25T19:42:08.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:08.530+0000 7f70b8f1c640 1 librados: shutdown 2024-03-25T19:42:08.541 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:15.348 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.464+0000 7fc2308fb640 10 monclient: get_monmap_and_config 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.464+0000 7fc2308fb640 10 monclient: build_initial_monmap 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.464+0000 7fc2308fb640 10 monclient: monmap: 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:42:15.465297+0000 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:42:15.465297+0000 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.464+0000 7fc2308fb640 1 Processor -- start 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.464+0000 7fc2308fb640 1 -- start start 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.464+0000 7fc2308fb640 10 monclient: init 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc22c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc22c13f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc22c1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c142520 con 0x7fc22c1430c0 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c0b28a0 con 0x7fc22c0a98f0 2024-03-25T19:42:15.467 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c0b2a30 con 0x7fc22c13f710 2024-03-25T19:42:15.467 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:15.467 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.465+0000 7fc2308fb640 10 monclient(hunting): authenticate will time out at 4947.734508s 2024-03-25T19:42:15.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.477+0000 7fc22a59c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc22c0a98f0 0x7fc22c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:32812/0 (socket says 172.21.15.69:32812) 2024-03-25T19:42:15.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.477+0000 7fc22a59c640 1 -- 172.21.15.69:0/3803410904 learned_addr learned my addr 172.21.15.69:0/3803410904 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:15.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.498+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248383199 0 0) 0x7fc22c0b28a0 con 0x7fc22c0a98f0 2024-03-25T19:42:15.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.498+0000 7fc229d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.498+0000 7fc229d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.498+0000 7fc229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.498+0000 7fc229d9b640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:42:15.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.499+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc218003520 con 0x7fc22c0a98f0 2024-03-25T19:42:15.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.504+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242595009 0 0) 0x7fc22c142520 con 0x7fc22c1430c0 2024-03-25T19:42:15.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.504+0000 7fc229d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.504+0000 7fc229d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.504+0000 7fc229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.504+0000 7fc229d9b640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:42:15.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.504+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc22c0b28a0 con 0x7fc22c1430c0 2024-03-25T19:42:15.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.507+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982664026 0 0) 0x7fc22c0b2a30 con 0x7fc22c13f710 2024-03-25T19:42:15.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.507+0000 7fc229d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.507+0000 7fc229d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.507+0000 7fc229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.507+0000 7fc229d9b640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:42:15.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.507+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc22c142520 con 0x7fc22c13f710 2024-03-25T19:42:15.514 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176922867 0 0) 0x7fc22c142520 con 0x7fc22c13f710 2024-03-25T19:42:15.514 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 >> v1:172.21.15.169:6789/0 conn(0x7fc22c0a98f0 legacy=0x7fc22c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 >> v1:172.21.15.69:6789/0 conn(0x7fc22c1430c0 legacy=0x7fc22c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 1 monclient: found mon.noname-b 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0b44c0 con 0x7fc22c13f710 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc2140015b0 con 0x7fc22c13f710 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc2308fb640 5 monclient: authenticate success, global_id 4277 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc2308fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc229d9b640 10 monclient: dump: 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:15.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.514+0000 7fc2308fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:15.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.517+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc214001710 con 0x7fc22c13f710 2024-03-25T19:42:15.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.517+0000 7fc229d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:15.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.517+0000 7fc229d9b640 1 -- 172.21.15.69:0/3803410904 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc214001b60 con 0x7fc22c13f710 2024-03-25T19:42:15.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.517+0000 7fc2308fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.517+0000 7fc2308fb640 4 set_mon_vals no callback set 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.517+0000 7fc229d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.517+0000 7fc229d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.517+0000 7fc229d9b640 10 monclient: dump: 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:15.518 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:15.519 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:15.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.518+0000 7fc2308fb640 10 monclient: shutdown 2024-03-25T19:42:15.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.518+0000 7fc2308fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:15.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.518+0000 7fc2308fb640 1 -- 172.21.15.69:0/3803410904 >> v1:172.21.15.69:6790/0 conn(0x7fc22c13f710 legacy=0x7fc22c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.518+0000 7fc2308fb640 1 -- 172.21.15.69:0/3803410904 shutdown_connections 2024-03-25T19:42:15.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.518+0000 7fc2308fb640 1 -- 172.21.15.69:0/3803410904 >> 172.21.15.69:0/3803410904 conn(0x7fc22c0b7ea0 msgr2=0x7fc22c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:15.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.518+0000 7fc2308fb640 1 -- 172.21.15.69:0/3803410904 shutdown_connections 2024-03-25T19:42:15.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.518+0000 7fc2308fb640 1 -- 172.21.15.69:0/3803410904 wait complete. 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.519+0000 7fc2308fb640 10 monclient: build_initial_monmap 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.519+0000 7fc2308fb640 10 monclient: monmap: 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:15.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.519+0000 7fc2308fb640 1 librados: starting msgr at 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.519+0000 7fc2308fb640 1 librados: starting objecter 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 1 Processor -- start 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 1 -- start start 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 1 librados: setting wanted keys 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 1 librados: calling monclient init 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 10 monclient: init 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc22c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc22c13f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc22c1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c1df980 con 0x7fc22c1430c0 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.520+0000 7fc2308fb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c14eda0 con 0x7fc22c0a98f0 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.521+0000 7fc2308fb640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c1df610 con 0x7fc22c13f710 2024-03-25T19:42:15.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.521+0000 7fc2308fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:15.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.521+0000 7fc2308fb640 10 monclient(hunting): authenticate will time out at 4947.789625s 2024-03-25T19:42:15.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.530+0000 7fc223fff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fc22c13f710 0x7fc22c138ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:49390/0 (socket says 172.21.15.69:49390) 2024-03-25T19:42:15.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.530+0000 7fc223fff640 1 -- 172.21.15.69:0/2626249578 learned_addr learned my addr 172.21.15.69:0/2626249578 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:15.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.531+0000 7fc222ffd640 10 client.?.objecter ms_handle_connect 0x7fc22c13f710 2024-03-25T19:42:15.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.531+0000 7fc222ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:15.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.541+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992008960 0 0) 0x7fc22c1df610 con 0x7fc22c13f710 2024-03-25T19:42:15.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.541+0000 7fc222ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.541+0000 7fc222ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.541+0000 7fc222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.541+0000 7fc222ffd640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:42:15.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.541+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc20c004eb0 con 0x7fc22c13f710 2024-03-25T19:42:15.545 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.543+0000 7fc222ffd640 10 client.?.objecter ms_handle_connect 0x7fc22c1430c0 2024-03-25T19:42:15.545 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.543+0000 7fc222ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:15.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.549+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437966622 0 0) 0x7fc22c1df980 con 0x7fc22c1430c0 2024-03-25T19:42:15.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.549+0000 7fc222ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.549+0000 7fc222ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.549+0000 7fc222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.549+0000 7fc222ffd640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:42:15.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.549+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc22c1df610 con 0x7fc22c1430c0 2024-03-25T19:42:15.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.550+0000 7fc222ffd640 10 client.?.objecter ms_handle_connect 0x7fc22c0a98f0 2024-03-25T19:42:15.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.550+0000 7fc222ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:15.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.562+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 291579429 0 0) 0x7fc20c004eb0 con 0x7fc22c13f710 2024-03-25T19:42:15.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.562+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc22c1df980 con 0x7fc22c13f710 2024-03-25T19:42:15.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.562+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc214001ae0 con 0x7fc22c13f710 2024-03-25T19:42:15.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.562+0000 7fc222ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.562+0000 7fc222ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.562+0000 7fc222ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:15.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.562+0000 7fc222ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:15.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.568+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736145162 0 0) 0x7fc22c14eda0 con 0x7fc22c0a98f0 2024-03-25T19:42:15.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.568+0000 7fc222ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.568+0000 7fc222ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.568+0000 7fc222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.568+0000 7fc222ffd640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:42:15.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.568+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc20c004eb0 con 0x7fc22c0a98f0 2024-03-25T19:42:15.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.577+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3257766123 0 0) 0x7fc22c1df610 con 0x7fc22c1430c0 2024-03-25T19:42:15.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.577+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc22c14eda0 con 0x7fc22c1430c0 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.577+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc224001bd0 con 0x7fc22c1430c0 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.577+0000 7fc222ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.577+0000 7fc222ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.577+0000 7fc222ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:15.578 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:15.579 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:15.579 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:15.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.578+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 912108651 0 0) 0x7fc22c1df980 con 0x7fc22c13f710 2024-03-25T19:42:15.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.578+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 >> v1:172.21.15.169:6789/0 conn(0x7fc22c0a98f0 legacy=0x7fc22c1def00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.578+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 >> v1:172.21.15.69:6789/0 conn(0x7fc22c1430c0 legacy=0x7fc22c0708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.578+0000 7fc222ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.578+0000 7fc222ffd640 1 monclient: found mon.c 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.578+0000 7fc222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.578+0000 7fc222ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.578+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c071e70 con 0x7fc22c13f710 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc222ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc222ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 5 monclient: authenticate success, global_id 4280 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 10 monclient: _renew_subs 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22c072080 con 0x7fc22c13f710 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 10 monclient: _renew_subs 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:15.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22c1398e0 con 0x7fc22c13f710 2024-03-25T19:42:15.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.579+0000 7fc2308fb640 1 librados: init done 2024-03-25T19:42:15.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.580+0000 7fc2308fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:15.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.580+0000 7fc2308fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:15.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.580+0000 7fc2308fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:15.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.580+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc22c0b3ef0 con 0x7fc22c13f710 2024-03-25T19:42:15.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.896+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2140038a0 con 0x7fc22c13f710 2024-03-25T19:42:15.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.896+0000 7fc222ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:15.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.896+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc214003b10 con 0x7fc22c13f710 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.896+0000 7fc222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.896+0000 7fc222ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.896+0000 7fc222ffd640 10 monclient: dump: 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:15.897 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.896+0000 7fc22959a640 4 set_mon_vals no callback set 2024-03-25T19:42:15.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.897+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fc214003d30 con 0x7fc22c13f710 2024-03-25T19:42:15.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.898+0000 7fc22a59c640 10 monclient: get_auth_request con 0x7fc20c06dfb0 auth_method 0 2024-03-25T19:42:15.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.898+0000 7fc222ffd640 10 client.4280.objecter ms_handle_connect 0x7fc20c06dfb0 2024-03-25T19:42:15.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.905+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fc2140546a0 con 0x7fc22c13f710 2024-03-25T19:42:15.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.905+0000 7fc222ffd640 10 client.4280.objecter ms_dispatch 0x7fc22c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:15.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.905+0000 7fc222ffd640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:15.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.905+0000 7fc222ffd640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:15.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.905+0000 7fc222ffd640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-25T19:42:15.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.905+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2140501b0 con 0x7fc22c13f710 2024-03-25T19:42:15.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.905+0000 7fc222ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:15.906 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:15.905+0000 7fc222ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:16.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.151+0000 7fc2308fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:16.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.152+0000 7fc2308fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:16.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.152+0000 7fc2308fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:16.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.152+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc22c1df7a0 con 0x7fc22c13f710 2024-03-25T19:42:16.185 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.185+0000 7fc222ffd640 1 -- 172.21.15.69:0/2626249578 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fc2140520e0 con 0x7fc22c13f710 2024-03-25T19:42:16.185 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.185+0000 7fc222ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:16.185 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.185+0000 7fc222ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:16.185 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:42:16.186 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:16.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 10 librados: watch_flush enter 2024-03-25T19:42:16.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 10 librados: watch_flush exit 2024-03-25T19:42:16.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-25T19:42:16.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 10 client.4280.objecter successfully canceled tick 2024-03-25T19:42:16.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 >> v1:172.21.15.169:6800/3072548259 conn(0x7fc20c06dfb0 legacy=0x7fc20c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:16.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 10 monclient: shutdown 2024-03-25T19:42:16.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:16.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 >> v1:172.21.15.69:6790/0 conn(0x7fc22c13f710 legacy=0x7fc22c138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:16.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 shutdown_connections 2024-03-25T19:42:16.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 >> 172.21.15.69:0/2626249578 conn(0x7fc22c0b7ea0 msgr2=0x7fc22c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:16.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 shutdown_connections 2024-03-25T19:42:16.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 1 -- 172.21.15.69:0/2626249578 wait complete. 2024-03-25T19:42:16.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:16.188+0000 7fc2308fb640 1 librados: shutdown 2024-03-25T19:42:16.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:21.385 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:59.349 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.436 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:59.400 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.470 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:59.434 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.481 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:59.445 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.482 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:59.436 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.489 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:59.443 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.491 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:59.455 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.523 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:59.487 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.524 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:59.479 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.006 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:23.123 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.123+0000 7fc6a42d9640 10 monclient: get_monmap_and_config 2024-03-25T19:42:23.123 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.123+0000 7fc6a42d9640 10 monclient: build_initial_monmap 2024-03-25T19:42:23.123 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.123+0000 7fc6a42d9640 10 monclient: monmap: 2024-03-25T19:42:23.123 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:42:23.124989+0000 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:42:23.124989+0000 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:42:23.124 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:23.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.124+0000 7fc6a42d9640 1 Processor -- start 2024-03-25T19:42:23.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.124+0000 7fc6a42d9640 1 -- start start 2024-03-25T19:42:23.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.124+0000 7fc6a42d9640 10 monclient: init 2024-03-25T19:42:23.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:23.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:23.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc69c0b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:23.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc69c0b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:23.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc69c14be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:23.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0af940 con 0x7fc69c0b1730 2024-03-25T19:42:23.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0b0a70 con 0x7fc69c0b4d60 2024-03-25T19:42:23.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0b0d50 con 0x7fc69c14be90 2024-03-25T19:42:23.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:23.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.125+0000 7fc6a42d9640 10 monclient(hunting): authenticate will time out at 4955.393850s 2024-03-25T19:42:23.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.131+0000 7fc6a2075640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc69c0b4d60 0x7fc69c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37600/0 (socket says 172.21.15.69:37600) 2024-03-25T19:42:23.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.131+0000 7fc6a2075640 1 -- 172.21.15.69:0/2597340394 learned_addr learned my addr 172.21.15.69:0/2597340394 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:23.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.145+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322429154 0 0) 0x7fc69c0b0a70 con 0x7fc69c0b4d60 2024-03-25T19:42:23.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.145+0000 7fc6a1073640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.145+0000 7fc6a1073640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.145+0000 7fc6a1073640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.145+0000 7fc6a1073640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:42:23.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.145+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc688003520 con 0x7fc69c0b4d60 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.158+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1996424873 0 0) 0x7fc688003520 con 0x7fc69c0b4d60 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.158+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 >> v1:172.21.15.69:6790/0 conn(0x7fc69c14be90 legacy=0x7fc69c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 >> v1:172.21.15.69:6789/0 conn(0x7fc69c0b1730 legacy=0x7fc69c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 10 monclient: _finish_hunting 0 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 1 monclient: found mon.noname-c 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c0b1030 con 0x7fc69c0b4d60 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 10 monclient: _finish_auth 0 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 10 monclient: _check_auth_tickets 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:23.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc698001690 con 0x7fc69c0b4d60 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a1073640 10 monclient: dump: 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a42d9640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:42:23.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.159+0000 7fc6a42d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:23.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.165+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6980017f0 con 0x7fc69c0b4d60 2024-03-25T19:42:23.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.165+0000 7fc6a1073640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.165+0000 7fc6a1073640 1 -- 172.21.15.69:0/2597340394 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc698001c40 con 0x7fc69c0b4d60 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.165+0000 7fc6a42d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.165+0000 7fc6a42d9640 4 set_mon_vals no callback set 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.165+0000 7fc6a1073640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.165+0000 7fc6a1073640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.165+0000 7fc6a1073640 10 monclient: dump: 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:23.166 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:23.167 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:23.167 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:23.167 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.166+0000 7fc6a42d9640 10 monclient: shutdown 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.166+0000 7fc6a42d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.166+0000 7fc6a42d9640 1 -- 172.21.15.69:0/2597340394 >> v1:172.21.15.169:6789/0 conn(0x7fc69c0b4d60 legacy=0x7fc69c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.166+0000 7fc6a42d9640 1 -- 172.21.15.69:0/2597340394 shutdown_connections 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.166+0000 7fc6a42d9640 1 -- 172.21.15.69:0/2597340394 >> 172.21.15.69:0/2597340394 conn(0x7fc69c0617a0 msgr2=0x7fc69c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.166+0000 7fc6a42d9640 1 -- 172.21.15.69:0/2597340394 shutdown_connections 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.166+0000 7fc6a42d9640 1 -- 172.21.15.69:0/2597340394 wait complete. 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 10 monclient: build_initial_monmap 2024-03-25T19:42:23.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 10 monclient: monmap: 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 1 librados: starting msgr at 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 1 librados: starting objecter 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 1 Processor -- start 2024-03-25T19:42:23.170 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 1 -- start start 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 1 librados: setting wanted keys 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 1 librados: calling monclient init 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.167+0000 7fc6a42d9640 10 monclient: init 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc69c0b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc69c0b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc69c14be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c13acc0 con 0x7fc69c0b4d60 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c1d8620 con 0x7fc69c14be90 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c13af30 con 0x7fc69c0b1730 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:23.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.168+0000 7fc6a42d9640 10 monclient(hunting): authenticate will time out at 4955.437591s 2024-03-25T19:42:23.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.178+0000 7fc6a2876640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc69c14be90 0x7fc69c1e30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37658/0 (socket says 172.21.15.69:37658) 2024-03-25T19:42:23.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.178+0000 7fc6a2876640 1 -- 172.21.15.69:0/1556737994 learned_addr learned my addr 172.21.15.69:0/1556737994 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:23.183 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.183+0000 7fc692ffd640 10 client.?.objecter ms_handle_connect 0x7fc69c14be90 2024-03-25T19:42:23.183 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.183+0000 7fc692ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:23.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.186+0000 7fc692ffd640 10 client.?.objecter ms_handle_connect 0x7fc69c0b4d60 2024-03-25T19:42:23.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.186+0000 7fc692ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:23.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.195+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105602978 0 0) 0x7fc69c13acc0 con 0x7fc69c0b4d60 2024-03-25T19:42:23.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.195+0000 7fc692ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.195+0000 7fc692ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.195+0000 7fc692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.195+0000 7fc692ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:42:23.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.195+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc680004eb0 con 0x7fc69c0b4d60 2024-03-25T19:42:23.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.197+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549426259 0 0) 0x7fc69c1d8620 con 0x7fc69c14be90 2024-03-25T19:42:23.197 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.197+0000 7fc692ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.197 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.197+0000 7fc692ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.197 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.197+0000 7fc692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.197 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.197+0000 7fc692ffd640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:42:23.197 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.197+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c13acc0 con 0x7fc69c14be90 2024-03-25T19:42:23.199 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.199+0000 7fc692ffd640 10 client.?.objecter ms_handle_connect 0x7fc69c0b1730 2024-03-25T19:42:23.199 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.199+0000 7fc692ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:23.210 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.210+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008205955 0 0) 0x7fc69c13af30 con 0x7fc69c0b1730 2024-03-25T19:42:23.210 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.210+0000 7fc692ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.210 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.210+0000 7fc692ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.210 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.210+0000 7fc692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.210 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.210+0000 7fc692ffd640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:42:23.210 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.210+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c1d8620 con 0x7fc69c0b1730 2024-03-25T19:42:23.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.212+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 727916885 0 0) 0x7fc69c13acc0 con 0x7fc69c14be90 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.213+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc69c13af30 con 0x7fc69c14be90 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.213+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc694001c50 con 0x7fc69c14be90 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.213+0000 7fc692ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.213+0000 7fc692ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.213+0000 7fc692ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:23.213 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:23.214 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:23.214 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:23.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.213+0000 7fc692ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:23.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.214+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102254681 0 0) 0x7fc680004eb0 con 0x7fc69c0b4d60 2024-03-25T19:42:23.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.214+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc69c13acc0 con 0x7fc69c0b4d60 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.214+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc68c001930 con 0x7fc69c0b4d60 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.214+0000 7fc692ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.214+0000 7fc692ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.214+0000 7fc692ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:23.215 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2747000981 0 0) 0x7fc69c13af30 con 0x7fc69c14be90 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 >> v1:172.21.15.69:6790/0 conn(0x7fc69c0b1730 legacy=0x7fc69c1d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 >> v1:172.21.15.69:6789/0 conn(0x7fc69c0b4d60 legacy=0x7fc69c13bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 1 monclient: found mon.b 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c1d79f0 con 0x7fc69c14be90 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:23.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:42:23.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 10 monclient: _renew_subs 2024-03-25T19:42:23.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:23.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c13c310 con 0x7fc69c14be90 2024-03-25T19:42:23.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:23.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 10 monclient: _renew_subs 2024-03-25T19:42:23.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:23.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c1394e0 con 0x7fc69c14be90 2024-03-25T19:42:23.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.220+0000 7fc6a42d9640 1 librados: init done 2024-03-25T19:42:23.222 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.221+0000 7fc6a42d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:23.222 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.221+0000 7fc6a42d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:23.222 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.221+0000 7fc6a42d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:23.222 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.221+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c1401f0 con 0x7fc69c14be90 2024-03-25T19:42:23.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.500+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc694003990 con 0x7fc69c14be90 2024-03-25T19:42:23.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.500+0000 7fc692ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.500+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc694003d60 con 0x7fc69c14be90 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.500+0000 7fc692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.500+0000 7fc692ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.500+0000 7fc692ffd640 10 monclient: dump: 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:23.501 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:23.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.500+0000 7fc6a0872640 4 set_mon_vals no callback set 2024-03-25T19:42:23.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.501+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fc694003f80 con 0x7fc69c14be90 2024-03-25T19:42:23.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.502+0000 7fc6a2075640 10 monclient: get_auth_request con 0x7fc68006df70 auth_method 0 2024-03-25T19:42:23.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.502+0000 7fc692ffd640 10 client.4282.objecter ms_handle_connect 0x7fc68006df70 2024-03-25T19:42:23.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.506+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fc694054a10 con 0x7fc69c14be90 2024-03-25T19:42:23.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.506+0000 7fc692ffd640 10 client.4282.objecter ms_dispatch 0x7fc69c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:23.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.506+0000 7fc692ffd640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:23.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.506+0000 7fc692ffd640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:23.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.506+0000 7fc692ffd640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T19:42:23.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.506+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc694050520 con 0x7fc69c14be90 2024-03-25T19:42:23.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.506+0000 7fc692ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:23.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.506+0000 7fc692ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:23.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.738+0000 7fc6a42d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:23.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.738+0000 7fc6a42d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:23.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.738+0000 7fc6a42d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:23.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.738+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc69c1d7860 con 0x7fc69c14be90 2024-03-25T19:42:23.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.758+0000 7fc692ffd640 1 -- 172.21.15.69:0/1556737994 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fc694052450 con 0x7fc69c14be90 2024-03-25T19:42:23.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.758+0000 7fc692ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:23.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.758+0000 7fc692ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:23.758 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:42:23.758 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 10 librados: watch_flush enter 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 10 librados: watch_flush exit 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 10 client.4282.objecter successfully canceled tick 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 >> v1:172.21.15.169:6800/3072548259 conn(0x7fc68006df70 legacy=0x7fc680070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 10 monclient: shutdown 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 >> v1:172.21.15.169:6789/0 conn(0x7fc69c14be90 legacy=0x7fc69c1e30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 shutdown_connections 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 >> 172.21.15.69:0/1556737994 conn(0x7fc69c0617a0 msgr2=0x7fc69c0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.760+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 shutdown_connections 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.761+0000 7fc6a42d9640 1 -- 172.21.15.69:0/1556737994 wait complete. 2024-03-25T19:42:23.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:23.761+0000 7fc6a42d9640 1 librados: shutdown 2024-03-25T19:42:23.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:30.579 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:30.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.691+0000 7f417d58d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:30.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.691+0000 7f417d58d640 10 monclient: build_initial_monmap 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.691+0000 7f417d58d640 10 monclient: monmap: 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:42:30.692918+0000 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:42:30.692918+0000 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:42:30.692 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.692+0000 7f417d58d640 1 Processor -- start 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.692+0000 7f417d58d640 1 -- start start 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.692+0000 7f417d58d640 10 monclient: init 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 10 monclient(hunting): picked mon.noname-b con 0x7f41780a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 10 monclient(hunting): picked mon.noname-c con 0x7f417813f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 10 monclient(hunting): picked mon.noname-a con 0x7f41781430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4178142520 con 0x7f41781430c0 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780b28a0 con 0x7f417813f710 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780b2a30 con 0x7f41780a98f0 2024-03-25T19:42:30.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:30.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f417d58d640 10 monclient(hunting): authenticate will time out at 4962.961732s 2024-03-25T19:42:30.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f416ffff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f417813f710 0x7f41781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:58010/0 (socket says 172.21.15.69:58010) 2024-03-25T19:42:30.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f4176ffd640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f41780a98f0 0x7f41780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:57964/0 (socket says 172.21.15.69:57964) 2024-03-25T19:42:30.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.693+0000 7f416ffff640 1 -- 172.21.15.69:0/1549173925 learned_addr learned my addr 172.21.15.69:0/1549173925 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:30.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.759+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060826857 0 0) 0x7f4178142520 con 0x7f41781430c0 2024-03-25T19:42:30.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.759+0000 7f41767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:30.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.759+0000 7f41767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:30.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.759+0000 7f41767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:30.760 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.759+0000 7f41767fc640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:42:30.760 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.760+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4164003520 con 0x7f41781430c0 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 363069972 0 0) 0x7f4164003520 con 0x7f41781430c0 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 >> v1:172.21.15.69:6790/0 conn(0x7f41780a98f0 legacy=0x7f41780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 >> v1:172.21.15.169:6789/0 conn(0x7f417813f710 legacy=0x7f41781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 1 monclient: found mon.noname-a 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41780b44c0 con 0x7f41781430c0 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4170001a10 con 0x7f41781430c0 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f417d58d640 5 monclient: authenticate success, global_id 4305 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f417d58d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:30.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f41767fc640 10 monclient: dump: 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:30.794 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.792+0000 7f417d58d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:30.816 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.816+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4170001b70 con 0x7f41781430c0 2024-03-25T19:42:30.816 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.816+0000 7f41767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:30.816 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.816+0000 7f41767fc640 1 -- 172.21.15.69:0/1549173925 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4170001f80 con 0x7f41781430c0 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.816+0000 7f417d58d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.816+0000 7f417d58d640 4 set_mon_vals no callback set 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.816+0000 7f41767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.816+0000 7f41767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.816+0000 7f41767fc640 10 monclient: dump: 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:30.817 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:30.818 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:30.818 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:30.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.817+0000 7f417d58d640 10 monclient: shutdown 2024-03-25T19:42:30.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.817+0000 7f417d58d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:30.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.817+0000 7f417d58d640 1 -- 172.21.15.69:0/1549173925 >> v1:172.21.15.69:6789/0 conn(0x7f41781430c0 legacy=0x7f417814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.817+0000 7f417d58d640 1 -- 172.21.15.69:0/1549173925 shutdown_connections 2024-03-25T19:42:30.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.817+0000 7f417d58d640 1 -- 172.21.15.69:0/1549173925 >> 172.21.15.69:0/1549173925 conn(0x7f41780b7ea0 msgr2=0x7f41780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:30.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.817+0000 7f417d58d640 1 -- 172.21.15.69:0/1549173925 shutdown_connections 2024-03-25T19:42:30.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.817+0000 7f417d58d640 1 -- 172.21.15.69:0/1549173925 wait complete. 2024-03-25T19:42:30.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 10 monclient: build_initial_monmap 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 10 monclient: monmap: 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 1 librados: starting msgr at 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 1 librados: starting objecter 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 1 Processor -- start 2024-03-25T19:42:30.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 1 -- start start 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 1 librados: setting wanted keys 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 1 librados: calling monclient init 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.818+0000 7f417d58d640 10 monclient: init 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 10 monclient(hunting): picked mon.noname-a con 0x7f41780a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 10 monclient(hunting): picked mon.noname-c con 0x7f417813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 10 monclient(hunting): picked mon.noname-b con 0x7f41781430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780abac0 con 0x7f41780a98f0 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780abda0 con 0x7f41781430c0 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41781380e0 con 0x7f417813f710 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.819+0000 7f417d58d640 10 monclient(hunting): authenticate will time out at 4963.088213s 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.820+0000 7f41777fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f41781430c0 0x7f41781e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:58072/0 (socket says 172.21.15.69:58072) 2024-03-25T19:42:30.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.820+0000 7f41777fe640 1 -- 172.21.15.69:0/3058254443 learned_addr learned my addr 172.21.15.69:0/3058254443 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:30.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.820+0000 7f416f7fe640 10 client.?.objecter ms_handle_connect 0x7f41781430c0 2024-03-25T19:42:30.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.820+0000 7f416f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:30.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.820+0000 7f416f7fe640 10 client.?.objecter ms_handle_connect 0x7f417813f710 2024-03-25T19:42:30.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.820+0000 7f416f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:30.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.829+0000 7f416f7fe640 10 client.?.objecter ms_handle_connect 0x7f41780a98f0 2024-03-25T19:42:30.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.829+0000 7f416f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:30.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.850+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775434480 0 0) 0x7f41780abac0 con 0x7f41780a98f0 2024-03-25T19:42:30.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.850+0000 7f416f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:30.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.850+0000 7f416f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:30.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.850+0000 7f416f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:30.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.850+0000 7f416f7fe640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:42:30.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.850+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4158004eb0 con 0x7f41780a98f0 2024-03-25T19:42:30.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.872+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797265997 0 0) 0x7f4158004eb0 con 0x7f41780a98f0 2024-03-25T19:42:30.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.872+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41780abac0 con 0x7f41780a98f0 2024-03-25T19:42:30.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.872+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f41680018c0 con 0x7f41780a98f0 2024-03-25T19:42:30.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.872+0000 7f416f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:30.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.872+0000 7f416f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.872+0000 7f416f7fe640 10 monclient(hunting): dump: 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:30.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.872+0000 7f416f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 147424317 0 0) 0x7f41780abac0 con 0x7f41780a98f0 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 >> v1:172.21.15.69:6790/0 conn(0x7f417813f710 legacy=0x7f4178138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 >> v1:172.21.15.169:6789/0 conn(0x7f41781430c0 legacy=0x7f41781e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 1 monclient: found mon.a 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178137d90 con 0x7f41780a98f0 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f416f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f417d58d640 5 monclient: authenticate success, global_id 4308 2024-03-25T19:42:30.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f417d58d640 10 monclient: _renew_subs 2024-03-25T19:42:30.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f417d58d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:42:30.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.910+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4178139240 con 0x7f41780a98f0 2024-03-25T19:42:30.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:30.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 10 monclient: _renew_subs 2024-03-25T19:42:30.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:42:30.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4178136830 con 0x7f41780a98f0 2024-03-25T19:42:30.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 1 librados: init done 2024-03-25T19:42:30.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:30.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:30.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:42:30.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:30.911+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4178136370 con 0x7f41780a98f0 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.204+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41680047e0 con 0x7f41780a98f0 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.204+0000 7f416f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.204+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4168004a10 con 0x7f41780a98f0 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.204+0000 7f416f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.204+0000 7f416f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.204+0000 7f416f7fe640 10 monclient: dump: 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:31.205 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:31.206 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:31.206 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:31.206 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:31.206 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:31.206 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:31.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.204+0000 7f4175ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:31.207 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.206+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f4168004c30 con 0x7f41780a98f0 2024-03-25T19:42:31.207 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.206+0000 7f416ffff640 10 monclient: get_auth_request con 0x7f415806dc50 auth_method 0 2024-03-25T19:42:31.207 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.207+0000 7f416f7fe640 10 client.4308.objecter ms_handle_connect 0x7f415806dc50 2024-03-25T19:42:31.210 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.210+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f4168054620 con 0x7f41780a98f0 2024-03-25T19:42:31.210 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.210+0000 7f416f7fe640 10 client.4308.objecter ms_dispatch 0x7f4178000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:31.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.210+0000 7f416f7fe640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:31.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.210+0000 7f416f7fe640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:31.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.210+0000 7f416f7fe640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-25T19:42:31.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.210+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4168050130 con 0x7f41780a98f0 2024-03-25T19:42:31.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.210+0000 7f416f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:31.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.210+0000 7f416f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:31.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.464+0000 7f417d58d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:31.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.464+0000 7f417d58d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:31.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.464+0000 7f417d58d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:42:31.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.464+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41780627b0 con 0x7f41780a98f0 2024-03-25T19:42:31.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.505+0000 7f416f7fe640 1 -- 172.21.15.69:0/3058254443 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f4168052060 con 0x7f41780a98f0 2024-03-25T19:42:31.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.505+0000 7f416f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:31.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.505+0000 7f416f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:31.505 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:42:31.505 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:31.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.507+0000 7f417d58d640 10 librados: watch_flush enter 2024-03-25T19:42:31.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.507+0000 7f417d58d640 10 librados: watch_flush exit 2024-03-25T19:42:31.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.507+0000 7f417d58d640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.507+0000 7f417d58d640 10 client.4308.objecter successfully canceled tick 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 >> v1:172.21.15.169:6800/3072548259 conn(0x7f415806dc50 legacy=0x7f4158070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 10 monclient: shutdown 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 >> v1:172.21.15.69:6789/0 conn(0x7f41780a98f0 legacy=0x7f41780ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 shutdown_connections 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 >> 172.21.15.69:0/3058254443 conn(0x7f41780b7ea0 msgr2=0x7f4178074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 shutdown_connections 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 1 -- 172.21.15.69:0/3058254443 wait complete. 2024-03-25T19:42:31.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:31.508+0000 7f417d58d640 1 librados: shutdown 2024-03-25T19:42:31.521 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:38.327 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:38.438 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.438+0000 7fd86ef0e640 10 monclient: get_monmap_and_config 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.438+0000 7fd86ef0e640 10 monclient: build_initial_monmap 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.438+0000 7fd86ef0e640 10 monclient: monmap: 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:42:38.440005+0000 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:42:38.440005+0000 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:42:38.439 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:38.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.439+0000 7fd86ef0e640 1 Processor -- start 2024-03-25T19:42:38.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.439+0000 7fd86ef0e640 1 -- start start 2024-03-25T19:42:38.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.439+0000 7fd86ef0e640 10 monclient: init 2024-03-25T19:42:38.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd8680a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd86813f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd8681430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd868142520 con 0x7fd8680a98f0 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8680b28a0 con 0x7fd86813f710 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8680b2a30 con 0x7fd8681430c0 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:38.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.440+0000 7fd86ef0e640 10 monclient(hunting): authenticate will time out at 4970.709217s 2024-03-25T19:42:38.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.446+0000 7fd86ccaa640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd8680a98f0 0x7fd8680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60342/0 (socket says 172.21.15.69:60342) 2024-03-25T19:42:38.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.446+0000 7fd86ccaa640 1 -- 172.21.15.69:0/3499713097 learned_addr learned my addr 172.21.15.69:0/3499713097 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:38.454 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.454+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152356338 0 0) 0x7fd868142520 con 0x7fd8680a98f0 2024-03-25T19:42:38.454 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.454+0000 7fd8677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:38.454 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.454+0000 7fd8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:38.454 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.454+0000 7fd8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:38.454 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.454+0000 7fd8677fe640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:42:38.454 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.454+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd858003520 con 0x7fd8680a98f0 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3129252935 0 0) 0x7fd858003520 con 0x7fd8680a98f0 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 >> v1:172.21.15.69:6790/0 conn(0x7fd8681430c0 legacy=0x7fd86814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 >> v1:172.21.15.169:6789/0 conn(0x7fd86813f710 legacy=0x7fd8681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 1 monclient: found mon.noname-a 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8680b44c0 con 0x7fd8680a98f0 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:38.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd8677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd86ef0e640 5 monclient: authenticate success, global_id 4311 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.461+0000 7fd86ef0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.462+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fd85c001690 con 0x7fd8680a98f0 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.462+0000 7fd8677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.462+0000 7fd8677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.462+0000 7fd8677fe640 10 monclient: dump: 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:38.462 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.463 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.463 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.463 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:38.463 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:38.463 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:38.463 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:38.463 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:38.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.462+0000 7fd86ef0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:38.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.475+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd85c001960 con 0x7fd8680a98f0 2024-03-25T19:42:38.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.475+0000 7fd8677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:38.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.475+0000 7fd8677fe640 1 -- 172.21.15.69:0/3499713097 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fd85c001cf0 con 0x7fd8680a98f0 2024-03-25T19:42:38.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.475+0000 7fd86ef0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:38.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.475+0000 7fd86ef0e640 4 set_mon_vals no callback set 2024-03-25T19:42:38.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.475+0000 7fd8677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.475 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.475+0000 7fd8677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.475+0000 7fd8677fe640 10 monclient: dump: 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:38.476 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:38.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.476+0000 7fd86ef0e640 10 monclient: shutdown 2024-03-25T19:42:38.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.476+0000 7fd86ef0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:38.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.476+0000 7fd86ef0e640 1 -- 172.21.15.69:0/3499713097 >> v1:172.21.15.69:6789/0 conn(0x7fd8680a98f0 legacy=0x7fd8680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.476+0000 7fd86ef0e640 1 -- 172.21.15.69:0/3499713097 shutdown_connections 2024-03-25T19:42:38.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.476+0000 7fd86ef0e640 1 -- 172.21.15.69:0/3499713097 >> 172.21.15.69:0/3499713097 conn(0x7fd8680b7ea0 msgr2=0x7fd8680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:38.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.476+0000 7fd86ef0e640 1 -- 172.21.15.69:0/3499713097 shutdown_connections 2024-03-25T19:42:38.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.477+0000 7fd86ef0e640 1 -- 172.21.15.69:0/3499713097 wait complete. 2024-03-25T19:42:38.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 10 monclient: build_initial_monmap 2024-03-25T19:42:38.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 10 monclient: monmap: 2024-03-25T19:42:38.478 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:38.478 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:38.478 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:42:38.478 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:42:38.478 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 1 librados: starting msgr at 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 1 librados: starting objecter 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 1 Processor -- start 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 1 -- start start 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 1 librados: setting wanted keys 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 1 librados: calling monclient init 2024-03-25T19:42:38.479 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.478+0000 7fd86ef0e640 10 monclient: init 2024-03-25T19:42:38.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:38.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:38.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd8680a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:38.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd86813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:38.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd8681430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:38.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd868137c40 con 0x7fd8680a98f0 2024-03-25T19:42:38.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd868138cb0 con 0x7fd8681430c0 2024-03-25T19:42:38.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd868139590 con 0x7fd86813f710 2024-03-25T19:42:38.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:38.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.479+0000 7fd86ef0e640 10 monclient(hunting): authenticate will time out at 4970.748108s 2024-03-25T19:42:38.491 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.491+0000 7fd86d4ab640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd8681430c0 0x7fd8681d76b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:34356/0 (socket says 172.21.15.69:34356) 2024-03-25T19:42:38.491 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.491+0000 7fd86d4ab640 1 -- 172.21.15.69:0/1491441037 learned_addr learned my addr 172.21.15.69:0/1491441037 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:38.491 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.492+0000 7fd8657fa640 10 client.?.objecter ms_handle_connect 0x7fd8681430c0 2024-03-25T19:42:38.492 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.492+0000 7fd8657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:38.492 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.492+0000 7fd8657fa640 10 client.?.objecter ms_handle_connect 0x7fd86813f710 2024-03-25T19:42:38.492 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.492+0000 7fd8657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:38.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.502+0000 7fd8657fa640 10 client.?.objecter ms_handle_connect 0x7fd8680a98f0 2024-03-25T19:42:38.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.502+0000 7fd8657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:38.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.504+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988868199 0 0) 0x7fd868139590 con 0x7fd86813f710 2024-03-25T19:42:38.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.504+0000 7fd8657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:38.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.504+0000 7fd8657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:38.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.504+0000 7fd8657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:38.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.504+0000 7fd8657fa640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:42:38.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.504+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd84c004eb0 con 0x7fd86813f710 2024-03-25T19:42:38.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.511+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906717722 0 0) 0x7fd868138cb0 con 0x7fd8681430c0 2024-03-25T19:42:38.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.511+0000 7fd8657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:38.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.511+0000 7fd8657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:38.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.511+0000 7fd8657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:38.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.511+0000 7fd8657fa640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:42:38.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.511+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd868139590 con 0x7fd8681430c0 2024-03-25T19:42:38.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.515+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418406858 0 0) 0x7fd868137c40 con 0x7fd8680a98f0 2024-03-25T19:42:38.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.515+0000 7fd8657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:38.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.515+0000 7fd8657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:38.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.515+0000 7fd8657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:38.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.515+0000 7fd8657fa640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:42:38.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.515+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd868138cb0 con 0x7fd8680a98f0 2024-03-25T19:42:38.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.521+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1360024866 0 0) 0x7fd84c004eb0 con 0x7fd86813f710 2024-03-25T19:42:38.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.521+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd868137c40 con 0x7fd86813f710 2024-03-25T19:42:38.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.521+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fd8540018f0 con 0x7fd86813f710 2024-03-25T19:42:38.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.521+0000 7fd8657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.521+0000 7fd8657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.521+0000 7fd8657fa640 10 monclient(hunting): dump: 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:38.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.521+0000 7fd8657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.529+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4022329499 0 0) 0x7fd868139590 con 0x7fd8681430c0 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.529+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd84c004eb0 con 0x7fd8681430c0 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.529+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fd860001a50 con 0x7fd8681430c0 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.529+0000 7fd8657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.529+0000 7fd8657fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.529+0000 7fd8657fa640 10 monclient(hunting): dump: 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.529 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.530 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.530 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:38.530 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:38.530 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:38.530 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:38.530 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:38.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2271025234 0 0) 0x7fd868137c40 con 0x7fd86813f710 2024-03-25T19:42:38.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 >> v1:172.21.15.169:6789/0 conn(0x7fd8681430c0 legacy=0x7fd8681d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 >> v1:172.21.15.69:6789/0 conn(0x7fd8680a98f0 legacy=0x7fd8681385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 1 monclient: found mon.c 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd868139870 con 0x7fd86813f710 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd8657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 5 monclient: authenticate success, global_id 4292 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 10 monclient: _renew_subs 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8681377d0 con 0x7fd86813f710 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 10 monclient: _renew_subs 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:38.533 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd868134900 con 0x7fd86813f710 2024-03-25T19:42:38.534 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.532+0000 7fd86ef0e640 1 librados: init done 2024-03-25T19:42:38.534 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.534+0000 7fd86ef0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:38.534 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.534+0000 7fd86ef0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:38.534 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.534+0000 7fd86ef0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:38.534 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.534+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8680b3ef0 con 0x7fd86813f710 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.835+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8540035f0 con 0x7fd86813f710 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.835+0000 7fd8657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.835+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fd8540039c0 con 0x7fd86813f710 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.835+0000 7fd8657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.835+0000 7fd8657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.835+0000 7fd8657fa640 10 monclient: dump: 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:38.835 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.836 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:38.836 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.836 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:38.836 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:38.836 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:38.836 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:38.836 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:38.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.835+0000 7fd866ffd640 4 set_mon_vals no callback set 2024-03-25T19:42:38.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.836+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fd854003be0 con 0x7fd86813f710 2024-03-25T19:42:38.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.837+0000 7fd86ccaa640 10 monclient: get_auth_request con 0x7fd84c06df70 auth_method 0 2024-03-25T19:42:38.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.837+0000 7fd8657fa640 10 client.4292.objecter ms_handle_connect 0x7fd84c06df70 2024-03-25T19:42:38.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.841+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fd8540535c0 con 0x7fd86813f710 2024-03-25T19:42:38.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.841+0000 7fd8657fa640 10 client.4292.objecter ms_dispatch 0x7fd868000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:38.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.841+0000 7fd8657fa640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:38.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.841+0000 7fd8657fa640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:38.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.841+0000 7fd8657fa640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T19:42:38.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.841+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd85404f0d0 con 0x7fd86813f710 2024-03-25T19:42:38.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.841+0000 7fd8657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:38.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:38.841+0000 7fd8657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:39.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.080+0000 7fd86ef0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:39.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.080+0000 7fd86ef0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:39.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.080+0000 7fd86ef0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:42:39.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.080+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8680627b0 con 0x7fd86813f710 2024-03-25T19:42:39.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.188+0000 7fd8657fa640 1 -- 172.21.15.69:0/1491441037 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fd854051000 con 0x7fd86813f710 2024-03-25T19:42:39.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.188+0000 7fd8657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:39.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.188+0000 7fd8657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:39.188 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:42:39.188 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:39.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.190+0000 7fd86ef0e640 10 librados: watch_flush enter 2024-03-25T19:42:39.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.190+0000 7fd86ef0e640 10 librados: watch_flush exit 2024-03-25T19:42:39.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.190+0000 7fd86ef0e640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T19:42:39.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.190+0000 7fd86ef0e640 10 client.4292.objecter successfully canceled tick 2024-03-25T19:42:39.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.190+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 >> v1:172.21.15.169:6800/3072548259 conn(0x7fd84c06df70 legacy=0x7fd84c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:39.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.190+0000 7fd86ef0e640 10 monclient: shutdown 2024-03-25T19:42:39.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.190+0000 7fd86ef0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:39.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.190+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 >> v1:172.21.15.69:6790/0 conn(0x7fd86813f710 legacy=0x7fd868134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:39.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.191+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 shutdown_connections 2024-03-25T19:42:39.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.191+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 >> 172.21.15.69:0/1491441037 conn(0x7fd8680b7ea0 msgr2=0x7fd868074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:39.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.191+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 shutdown_connections 2024-03-25T19:42:39.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.191+0000 7fd86ef0e640 1 -- 172.21.15.69:0/1491441037 wait complete. 2024-03-25T19:42:39.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:39.191+0000 7fd86ef0e640 1 librados: shutdown 2024-03-25T19:42:39.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:41.484 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:19.448 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.531 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:19.485 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.534 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:19.488 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.535 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:19.499 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.568 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:19.522 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.570 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:19.535 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.598 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:19.563 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.609 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:19.573 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.650 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:19.615 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:46.010 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:46.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.128+0000 7ff7b090e640 10 monclient: get_monmap_and_config 2024-03-25T19:42:46.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.128+0000 7ff7b090e640 10 monclient: build_initial_monmap 2024-03-25T19:42:46.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.128+0000 7ff7b090e640 10 monclient: monmap: 2024-03-25T19:42:46.127 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:46.127 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:46.127 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:42:46.128433+0000 2024-03-25T19:42:46.128 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:42:46.128433+0000 2024-03-25T19:42:46.128 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:46.128 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:46.128 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:46.128 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:42:46.128 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:42:46.128 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:46.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 1 Processor -- start 2024-03-25T19:42:46.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 1 -- start start 2024-03-25T19:42:46.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 10 monclient: init 2024-03-25T19:42:46.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:46.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:46.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 10 monclient(hunting): picked mon.noname-b con 0x7ff7ac0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:46.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 10 monclient(hunting): picked mon.noname-a con 0x7ff7ac13f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:46.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 10 monclient(hunting): picked mon.noname-c con 0x7ff7ac1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:46.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ac142520 con 0x7ff7ac13f710 2024-03-25T19:42:46.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ac0b28a0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ac0b2a30 con 0x7ff7ac0a98f0 2024-03-25T19:42:46.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:46.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.129+0000 7ff7b090e640 10 monclient(hunting): authenticate will time out at 4978.397324s 2024-03-25T19:42:46.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.136+0000 7ff7aad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ff7ac1430c0 0x7ff7ac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:47182/0 (socket says 172.21.15.69:47182) 2024-03-25T19:42:46.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.136+0000 7ff7aad9d640 1 -- 172.21.15.69:0/3714197110 learned_addr learned my addr 172.21.15.69:0/3714197110 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:46.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.143+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311789945 0 0) 0x7ff7ac0b28a0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.143+0000 7ff7a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.143+0000 7ff7a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.143+0000 7ff7a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.143+0000 7ff7a9d9b640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:42:46.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.143+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff798003520 con 0x7ff7ac1430c0 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.151+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3527513080 0 0) 0x7ff798003520 con 0x7ff7ac1430c0 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.151+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 >> v1:172.21.15.69:6790/0 conn(0x7ff7ac0a98f0 legacy=0x7ff7ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.151+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 >> v1:172.21.15.69:6789/0 conn(0x7ff7ac13f710 legacy=0x7ff7ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 1 monclient: found mon.noname-c 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac0b44c0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff7a4001a10 con 0x7ff7ac1430c0 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:46.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7a9d9b640 10 monclient: dump: 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7b090e640 5 monclient: authenticate success, global_id 4291 2024-03-25T19:42:46.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.152+0000 7ff7b090e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:46.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.162+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7a4001bf0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.162+0000 7ff7a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:46.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.162+0000 7ff7a9d9b640 1 -- 172.21.15.69:0/3714197110 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff7a4002040 con 0x7ff7ac1430c0 2024-03-25T19:42:46.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.162+0000 7ff7a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.162+0000 7ff7a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:46.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.162+0000 7ff7a9d9b640 10 monclient: dump: 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.162+0000 7ff7b090e640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:46.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.162+0000 7ff7b090e640 4 set_mon_vals no callback set 2024-03-25T19:42:46.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.163+0000 7ff7b090e640 10 monclient: shutdown 2024-03-25T19:42:46.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.163+0000 7ff7b090e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:46.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.163+0000 7ff7b090e640 1 -- 172.21.15.69:0/3714197110 >> v1:172.21.15.169:6789/0 conn(0x7ff7ac1430c0 legacy=0x7ff7ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.163+0000 7ff7b090e640 1 -- 172.21.15.69:0/3714197110 shutdown_connections 2024-03-25T19:42:46.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.163+0000 7ff7b090e640 1 -- 172.21.15.69:0/3714197110 >> 172.21.15.69:0/3714197110 conn(0x7ff7ac0b7ea0 msgr2=0x7ff7ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.163+0000 7ff7b090e640 1 -- 172.21.15.69:0/3714197110 shutdown_connections 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.164+0000 7ff7b090e640 1 -- 172.21.15.69:0/3714197110 wait complete. 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.164+0000 7ff7b090e640 10 monclient: build_initial_monmap 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.164+0000 7ff7b090e640 10 monclient: monmap: 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:42:46.164 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:46.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.165+0000 7ff7b090e640 1 librados: starting msgr at 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.165+0000 7ff7b090e640 1 librados: starting objecter 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.165+0000 7ff7b090e640 1 Processor -- start 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.165+0000 7ff7b090e640 1 -- start start 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.165+0000 7ff7b090e640 1 librados: setting wanted keys 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.165+0000 7ff7b090e640 1 librados: calling monclient init 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.165+0000 7ff7b090e640 10 monclient: init 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 10 monclient(hunting): picked mon.noname-a con 0x7ff7ac0a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 10 monclient(hunting): picked mon.noname-b con 0x7ff7ac13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 10 monclient(hunting): picked mon.noname-c con 0x7ff7ac1430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ac1d7670 con 0x7ff7ac0a98f0 2024-03-25T19:42:46.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ac0abdc0 con 0x7ff7ac13f710 2024-03-25T19:42:46.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ac138cc0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:46.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.166+0000 7ff7b090e640 10 monclient(hunting): authenticate will time out at 4978.434050s 2024-03-25T19:42:46.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.172+0000 7ff7aad9d640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7ff7ac1430c0 0x7ff7ac1d4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:46994/0 (socket says 172.21.15.69:46994) 2024-03-25T19:42:46.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.172+0000 7ff7aad9d640 1 -- 172.21.15.69:0/1044107899 learned_addr learned my addr 172.21.15.69:0/1044107899 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:46.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.173+0000 7ff7a2ffd640 10 client.?.objecter ms_handle_connect 0x7ff7ac1430c0 2024-03-25T19:42:46.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.173+0000 7ff7a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:46.173 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.175+0000 7ff7a2ffd640 10 client.?.objecter ms_handle_connect 0x7ff7ac13f710 2024-03-25T19:42:46.173 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.175+0000 7ff7a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:46.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.178+0000 7ff7a2ffd640 10 client.?.objecter ms_handle_connect 0x7ff7ac0a98f0 2024-03-25T19:42:46.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.178+0000 7ff7a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:46.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.181+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261076332 0 0) 0x7ff7ac0abdc0 con 0x7ff7ac13f710 2024-03-25T19:42:46.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.181+0000 7ff7a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.181+0000 7ff7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.181+0000 7ff7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.181+0000 7ff7a2ffd640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:42:46.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.181+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff78c004eb0 con 0x7ff7ac13f710 2024-03-25T19:42:46.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.185+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936893825 0 0) 0x7ff7ac138cc0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.185+0000 7ff7a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.185+0000 7ff7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.185+0000 7ff7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.185+0000 7ff7a2ffd640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:42:46.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.185+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7ac0abdc0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.190+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405749748 0 0) 0x7ff7ac1d7670 con 0x7ff7ac0a98f0 2024-03-25T19:42:46.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.190+0000 7ff7a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.190+0000 7ff7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.190+0000 7ff7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.190+0000 7ff7a2ffd640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:42:46.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.190+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7ac138cc0 con 0x7ff7ac0a98f0 2024-03-25T19:42:46.194 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.195+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238249579 0 0) 0x7ff78c004eb0 con 0x7ff7ac13f710 2024-03-25T19:42:46.194 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.195+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7ac1d7670 con 0x7ff7ac13f710 2024-03-25T19:42:46.194 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.195+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff794001930 con 0x7ff7ac13f710 2024-03-25T19:42:46.194 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.195+0000 7ff7a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.194 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.195+0000 7ff7a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:46.194 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.195+0000 7ff7a2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:46.194 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:46.194 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:46.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.195+0000 7ff7a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:46.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.214+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1123791445 0 0) 0x7ff7ac0abdc0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.214+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff78c004eb0 con 0x7ff7ac1430c0 2024-03-25T19:42:46.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.214+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff7a4001f00 con 0x7ff7ac1430c0 2024-03-25T19:42:46.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.214+0000 7ff7a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.214+0000 7ff7a2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:46.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.214+0000 7ff7a2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:46.213 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:46.214 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:46.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.216+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2055579314 0 0) 0x7ff7ac138cc0 con 0x7ff7ac0a98f0 2024-03-25T19:42:46.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.216+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7ac0abdc0 con 0x7ff7ac0a98f0 2024-03-25T19:42:46.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.216+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff79c001860 con 0x7ff7ac0a98f0 2024-03-25T19:42:46.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.216+0000 7ff7a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.216+0000 7ff7a2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:46.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.216+0000 7ff7a2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:46.215 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:46.216 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1007856958 0 0) 0x7ff7ac1d7670 con 0x7ff7ac13f710 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 >> v1:172.21.15.69:6790/0 conn(0x7ff7ac1430c0 legacy=0x7ff7ac1d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 >> v1:172.21.15.69:6789/0 conn(0x7ff7ac0a98f0 legacy=0x7ff7ac1385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 1 monclient: found mon.b 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac1d78a0 con 0x7ff7ac13f710 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:46.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 10 monclient: _renew_subs 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ac1d84d0 con 0x7ff7ac13f710 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 10 monclient: _renew_subs 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ac1d5210 con 0x7ff7ac13f710 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.217+0000 7ff7b090e640 1 librados: init done 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.218+0000 7ff7b090e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.218+0000 7ff7b090e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.218+0000 7ff7b090e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:46.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.218+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ac1d7ae0 con 0x7ff7ac13f710 2024-03-25T19:42:46.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.507+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7940035f0 con 0x7ff7ac13f710 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.507+0000 7ff7a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.507+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff7940037e0 con 0x7ff7ac13f710 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.507+0000 7ff7a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.507+0000 7ff7a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.507+0000 7ff7a2ffd640 10 monclient: dump: 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:46.507 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:46.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.507+0000 7ff7a959a640 4 set_mon_vals no callback set 2024-03-25T19:42:46.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.508+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7ff794003a00 con 0x7ff7ac13f710 2024-03-25T19:42:46.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.509+0000 7ff7aa59c640 10 monclient: get_auth_request con 0x7ff78c06e0f0 auth_method 0 2024-03-25T19:42:46.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.509+0000 7ff7a2ffd640 10 client.4294.objecter ms_handle_connect 0x7ff78c06e0f0 2024-03-25T19:42:46.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.514+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7ff794053490 con 0x7ff7ac13f710 2024-03-25T19:42:46.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.514+0000 7ff7a2ffd640 10 client.4294.objecter ms_dispatch 0x7ff7ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:46.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.514+0000 7ff7a2ffd640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:46.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.514+0000 7ff7a2ffd640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:46.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.514+0000 7ff7a2ffd640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T19:42:46.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.514+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff79404efa0 con 0x7ff7ac13f710 2024-03-25T19:42:46.514 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.514+0000 7ff7a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:46.514 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.515+0000 7ff7a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:46.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.739+0000 7ff7b090e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:46.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.739+0000 7ff7b090e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:46.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.739+0000 7ff7b090e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:46.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.739+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7ac0627b0 con 0x7ff7ac13f710 2024-03-25T19:42:46.861 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.862+0000 7ff7a2ffd640 1 -- 172.21.15.69:0/1044107899 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7ff794050ed0 con 0x7ff7ac13f710 2024-03-25T19:42:46.861 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.862+0000 7ff7a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:46.861 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.862+0000 7ff7a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:46.862 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:42:46.862 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.864+0000 7ff7b090e640 10 librados: watch_flush enter 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.864+0000 7ff7b090e640 10 librados: watch_flush exit 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.864+0000 7ff7b090e640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.864+0000 7ff7b090e640 10 client.4294.objecter successfully canceled tick 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.864+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 >> v1:172.21.15.169:6800/3072548259 conn(0x7ff78c06e0f0 legacy=0x7ff78c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.865+0000 7ff7b090e640 10 monclient: shutdown 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.865+0000 7ff7b090e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.865+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 >> v1:172.21.15.169:6789/0 conn(0x7ff7ac13f710 legacy=0x7ff7ac1d7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.865+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 shutdown_connections 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.865+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 >> 172.21.15.69:0/1044107899 conn(0x7ff7ac0b7ea0 msgr2=0x7ff7ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.865+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 shutdown_connections 2024-03-25T19:42:46.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.865+0000 7ff7b090e640 1 -- 172.21.15.69:0/1044107899 wait complete. 2024-03-25T19:42:46.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:46.865+0000 7ff7b090e640 1 librados: shutdown 2024-03-25T19:42:46.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:53.683 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:53.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.797+0000 7fc6edfa8640 10 monclient: get_monmap_and_config 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.797+0000 7fc6edfa8640 10 monclient: build_initial_monmap 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.797+0000 7fc6edfa8640 10 monclient: monmap: 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:42:53.798253+0000 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:42:53.798253+0000 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:42:53.797 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.797+0000 7fc6edfa8640 1 Processor -- start 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.797+0000 7fc6edfa8640 1 -- start start 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.797+0000 7fc6edfa8640 10 monclient: init 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e80a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:53.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e8142520 con 0x7fc6e80a98f0 2024-03-25T19:42:53.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e80b28a0 con 0x7fc6e81430c0 2024-03-25T19:42:53.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e80b2a30 con 0x7fc6e813f710 2024-03-25T19:42:53.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:53.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.798+0000 7fc6edfa8640 10 monclient(hunting): authenticate will time out at 4986.067520s 2024-03-25T19:42:53.805 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.805+0000 7fc6e7fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc6e81430c0 0x7fc6e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:35754/0 (socket says 172.21.15.69:35754) 2024-03-25T19:42:53.805 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.805+0000 7fc6e7fff640 1 -- 172.21.15.69:0/4206420651 learned_addr learned my addr 172.21.15.69:0/4206420651 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:53.818 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.818+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705507101 0 0) 0x7fc6e80b2a30 con 0x7fc6e813f710 2024-03-25T19:42:53.818 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.818+0000 7fc6e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.818 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.818+0000 7fc6e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.818 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.818+0000 7fc6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.818 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.818+0000 7fc6e67fc640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:42:53.818 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.818+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d0003520 con 0x7fc6e813f710 2024-03-25T19:42:53.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.825+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213445343 0 0) 0x7fc6e80b28a0 con 0x7fc6e81430c0 2024-03-25T19:42:53.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.825+0000 7fc6e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.825+0000 7fc6e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.825+0000 7fc6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.825+0000 7fc6e67fc640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:42:53.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.825+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e80b2a30 con 0x7fc6e81430c0 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3782015247 0 0) 0x7fc6d0003520 con 0x7fc6e813f710 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 >> v1:172.21.15.169:6789/0 conn(0x7fc6e81430c0 legacy=0x7fc6e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 >> v1:172.21.15.69:6789/0 conn(0x7fc6e80a98f0 legacy=0x7fc6e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 1 monclient: found mon.noname-b 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e80b44c0 con 0x7fc6e813f710 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc6dc001690 con 0x7fc6e813f710 2024-03-25T19:42:53.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.830+0000 7fc6e67fc640 10 monclient: dump: 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.831+0000 7fc6edfa8640 5 monclient: authenticate success, global_id 4301 2024-03-25T19:42:53.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.831+0000 7fc6edfa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.840+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6dc0017f0 con 0x7fc6e813f710 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.840+0000 7fc6e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.840+0000 7fc6e67fc640 1 -- 172.21.15.69:0/4206420651 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc6dc001c00 con 0x7fc6e813f710 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.840+0000 7fc6e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.840+0000 7fc6e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.840+0000 7fc6e67fc640 10 monclient: dump: 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:53.840 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:53.841 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:53.841 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:53.841 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:53.841 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:53.841 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:53.841 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:53.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.840+0000 7fc6edfa8640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:53.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.840+0000 7fc6edfa8640 4 set_mon_vals no callback set 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.841+0000 7fc6edfa8640 10 monclient: shutdown 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.841+0000 7fc6edfa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.841+0000 7fc6edfa8640 1 -- 172.21.15.69:0/4206420651 >> v1:172.21.15.69:6790/0 conn(0x7fc6e813f710 legacy=0x7fc6e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.841+0000 7fc6edfa8640 1 -- 172.21.15.69:0/4206420651 shutdown_connections 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.841+0000 7fc6edfa8640 1 -- 172.21.15.69:0/4206420651 >> 172.21.15.69:0/4206420651 conn(0x7fc6e80b7ea0 msgr2=0x7fc6e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.841+0000 7fc6edfa8640 1 -- 172.21.15.69:0/4206420651 shutdown_connections 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.841+0000 7fc6edfa8640 1 -- 172.21.15.69:0/4206420651 wait complete. 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 10 monclient: build_initial_monmap 2024-03-25T19:42:53.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 10 monclient: monmap: 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 1 librados: starting msgr at 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 1 librados: starting objecter 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 1 Processor -- start 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 1 -- start start 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 1 librados: setting wanted keys 2024-03-25T19:42:53.844 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 1 librados: calling monclient init 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.842+0000 7fc6edfa8640 10 monclient: init 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e80a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e8070f20 con 0x7fc6e813f710 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e806f140 con 0x7fc6e81430c0 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e806f2d0 con 0x7fc6e80a98f0 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:53.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.843+0000 7fc6edfa8640 10 monclient(hunting): authenticate will time out at 4986.112105s 2024-03-25T19:42:53.851 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.851+0000 7fc6e7fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc6e81430c0 0x7fc6e806ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:35780/0 (socket says 172.21.15.69:35780) 2024-03-25T19:42:53.851 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.851+0000 7fc6e7fff640 1 -- 172.21.15.69:0/2142027247 learned_addr learned my addr 172.21.15.69:0/2142027247 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:42:53.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.852+0000 7fc6cffff640 10 client.?.objecter ms_handle_connect 0x7fc6e81430c0 2024-03-25T19:42:53.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.852+0000 7fc6cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:53.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.855+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549826224 0 0) 0x7fc6e806f140 con 0x7fc6e81430c0 2024-03-25T19:42:53.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.855+0000 7fc6cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.855+0000 7fc6cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.855+0000 7fc6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.855+0000 7fc6cffff640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:42:53.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.855+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d4004eb0 con 0x7fc6e81430c0 2024-03-25T19:42:53.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.857+0000 7fc6cffff640 10 client.?.objecter ms_handle_connect 0x7fc6e80a98f0 2024-03-25T19:42:53.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.857+0000 7fc6cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:53.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.862+0000 7fc6cffff640 10 client.?.objecter ms_handle_connect 0x7fc6e813f710 2024-03-25T19:42:53.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.862+0000 7fc6cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:53.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.864+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183998531 0 0) 0x7fc6d4004eb0 con 0x7fc6e81430c0 2024-03-25T19:42:53.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.864+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6e806f140 con 0x7fc6e81430c0 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.865+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc6e0001d10 con 0x7fc6e81430c0 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.865+0000 7fc6cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.865+0000 7fc6cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.865+0000 7fc6cffff640 10 monclient(hunting): dump: 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:53.865 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:53.866 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:53.866 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:53.866 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:53.866 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:53.866 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:53.866 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.865+0000 7fc6cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:53.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.867+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448069699 0 0) 0x7fc6e8070f20 con 0x7fc6e813f710 2024-03-25T19:42:53.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.867+0000 7fc6cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.867+0000 7fc6cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.867+0000 7fc6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.867+0000 7fc6cffff640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:42:53.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.867+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d4004eb0 con 0x7fc6e813f710 2024-03-25T19:42:53.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710878116 0 0) 0x7fc6e806f2d0 con 0x7fc6e80a98f0 2024-03-25T19:42:53.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:42:53.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e8070f20 con 0x7fc6e80a98f0 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 698807728 0 0) 0x7fc6e806f140 con 0x7fc6e81430c0 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 >> v1:172.21.15.69:6790/0 conn(0x7fc6e80a98f0 legacy=0x7fc6e80713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 >> v1:172.21.15.69:6789/0 conn(0x7fc6e813f710 legacy=0x7fc6e8071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 1 monclient: found mon.b 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e806f460 con 0x7fc6e81430c0 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 10 monclient: _finish_auth 0 2024-03-25T19:42:53.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 10 monclient: _renew_subs 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e806f710 con 0x7fc6e81430c0 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 10 monclient: _renew_subs 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e81e78c0 con 0x7fc6e81430c0 2024-03-25T19:42:53.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.870+0000 7fc6edfa8640 1 librados: init done 2024-03-25T19:42:53.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.872+0000 7fc6edfa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:53.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.872+0000 7fc6edfa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:53.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.872+0000 7fc6edfa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:53.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:53.872+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e813e040 con 0x7fc6e81430c0 2024-03-25T19:42:54.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.152+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6e0004b60 con 0x7fc6e81430c0 2024-03-25T19:42:54.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.152+0000 7fc6cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.152+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc6e0004e70 con 0x7fc6e81430c0 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.152+0000 7fc6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.152+0000 7fc6cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.152+0000 7fc6cffff640 10 monclient: dump: 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:42:54.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.152+0000 7fc6e5ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:54.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.153+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fc6e001ae40 con 0x7fc6e81430c0 2024-03-25T19:42:54.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.154+0000 7fc6e77fe640 10 monclient: get_auth_request con 0x7fc6d406ddb0 auth_method 0 2024-03-25T19:42:54.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.154+0000 7fc6cffff640 10 client.4300.objecter ms_handle_connect 0x7fc6d406ddb0 2024-03-25T19:42:54.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.158+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fc6e0054bd0 con 0x7fc6e81430c0 2024-03-25T19:42:54.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.158+0000 7fc6cffff640 10 client.4300.objecter ms_dispatch 0x7fc6e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:54.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.158+0000 7fc6cffff640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:54.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.158+0000 7fc6cffff640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:54.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.158+0000 7fc6cffff640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T19:42:54.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.158+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6e0050510 con 0x7fc6e81430c0 2024-03-25T19:42:54.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.158+0000 7fc6cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:54.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.158+0000 7fc6cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.418+0000 7fc6edfa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.418+0000 7fc6edfa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.418+0000 7fc6edfa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.418+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6e80627b0 con 0x7fc6e81430c0 2024-03-25T19:42:54.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.439+0000 7fc6cffff640 1 -- 172.21.15.69:0/2142027247 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fc6e0052610 con 0x7fc6e81430c0 2024-03-25T19:42:54.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.439+0000 7fc6cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:54.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.439+0000 7fc6cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:54.440 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:42:54.440 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 10 librados: watch_flush enter 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 10 librados: watch_flush exit 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 10 client.4300.objecter successfully canceled tick 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 >> v1:172.21.15.169:6800/3072548259 conn(0x7fc6d406ddb0 legacy=0x7fc6d4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 10 monclient: shutdown 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 >> v1:172.21.15.169:6789/0 conn(0x7fc6e81430c0 legacy=0x7fc6e806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:54.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 shutdown_connections 2024-03-25T19:42:54.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 >> 172.21.15.69:0/2142027247 conn(0x7fc6e80b7ea0 msgr2=0x7fc6e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:54.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 shutdown_connections 2024-03-25T19:42:54.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.442+0000 7fc6edfa8640 1 -- 172.21.15.69:0/2142027247 wait complete. 2024-03-25T19:42:54.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:42:54.443+0000 7fc6edfa8640 1 librados: shutdown 2024-03-25T19:42:54.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:01.261 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:01.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.377+0000 7f8fe740d640 10 monclient: get_monmap_and_config 2024-03-25T19:43:01.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.377+0000 7f8fe740d640 10 monclient: build_initial_monmap 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.377+0000 7f8fe740d640 10 monclient: monmap: 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:43:01.378788+0000 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:43:01.378788+0000 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.378+0000 7f8fe740d640 1 Processor -- start 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.378+0000 7f8fe740d640 1 -- start start 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.378+0000 7f8fe740d640 10 monclient: init 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe00a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe013f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe0142520 con 0x7f8fe00a98f0 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00b28a0 con 0x7f8fe01430c0 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00b2a30 con 0x7f8fe013f710 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.379+0000 7f8fe740d640 10 monclient(hunting): authenticate will time out at 4993.647980s 2024-03-25T19:43:01.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.388+0000 7f8fe49a8640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f8fe013f710 0x7f8fe01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:51436/0 (socket says 172.21.15.69:51436) 2024-03-25T19:43:01.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.388+0000 7f8fe49a8640 1 -- 172.21.15.69:0/4256777191 learned_addr learned my addr 172.21.15.69:0/4256777191 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:01.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.401+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859205492 0 0) 0x7f8fe00b2a30 con 0x7f8fe013f710 2024-03-25T19:43:01.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.401+0000 7f8fd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.401+0000 7f8fd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.401+0000 7f8fd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.401+0000 7f8fd7fff640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:43:01.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.401+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fcc003520 con 0x7f8fe013f710 2024-03-25T19:43:01.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.402+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616797996 0 0) 0x7f8fe0142520 con 0x7f8fe00a98f0 2024-03-25T19:43:01.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.402+0000 7f8fd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.402+0000 7f8fd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.402+0000 7f8fd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.402+0000 7f8fd7fff640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.402+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe00b2a30 con 0x7f8fe00a98f0 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.403+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740031459 0 0) 0x7f8fe00b28a0 con 0x7f8fe01430c0 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.403+0000 7f8fd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.403+0000 7f8fd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.403+0000 7f8fd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.403+0000 7f8fd7fff640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:43:01.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.403+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe0142520 con 0x7f8fe01430c0 2024-03-25T19:43:01.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.416+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 876189811 0 0) 0x7f8fcc003520 con 0x7f8fe013f710 2024-03-25T19:43:01.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.416+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 >> v1:172.21.15.169:6789/0 conn(0x7f8fe01430c0 legacy=0x7f8fe014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.416+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 >> v1:172.21.15.69:6789/0 conn(0x7f8fe00a98f0 legacy=0x7f8fe00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:01.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 1 monclient: found mon.noname-b 2024-03-25T19:43:01.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe00b44c0 con 0x7f8fe013f710 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8fd0001690 con 0x7f8fe013f710 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fd7fff640 10 monclient: dump: 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.418 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.419 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:01.419 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:01.419 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:01.419 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:01.419 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:01.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fe740d640 5 monclient: authenticate success, global_id 4307 2024-03-25T19:43:01.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.417+0000 7f8fe740d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:01.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.433+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fd00017f0 con 0x7f8fe013f710 2024-03-25T19:43:01.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.433+0000 7f8fd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:01.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.433+0000 7f8fd7fff640 1 -- 172.21.15.69:0/4256777191 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8fd0001c00 con 0x7f8fe013f710 2024-03-25T19:43:01.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.433+0000 7f8fe740d640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.433+0000 7f8fe740d640 4 set_mon_vals no callback set 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.433+0000 7f8fd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.433+0000 7f8fd7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.433+0000 7f8fd7fff640 10 monclient: dump: 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:01.435 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:01.435 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:01.435 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:01.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.434+0000 7f8fe740d640 10 monclient: shutdown 2024-03-25T19:43:01.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.434+0000 7f8fe740d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:01.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.434+0000 7f8fe740d640 1 -- 172.21.15.69:0/4256777191 >> v1:172.21.15.69:6790/0 conn(0x7f8fe013f710 legacy=0x7f8fe01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.434+0000 7f8fe740d640 1 -- 172.21.15.69:0/4256777191 shutdown_connections 2024-03-25T19:43:01.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.434+0000 7f8fe740d640 1 -- 172.21.15.69:0/4256777191 >> 172.21.15.69:0/4256777191 conn(0x7f8fe00b7ea0 msgr2=0x7f8fe00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:01.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.434+0000 7f8fe740d640 1 -- 172.21.15.69:0/4256777191 shutdown_connections 2024-03-25T19:43:01.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.434+0000 7f8fe740d640 1 -- 172.21.15.69:0/4256777191 wait complete. 2024-03-25T19:43:01.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 10 monclient: build_initial_monmap 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 10 monclient: monmap: 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 1 librados: starting msgr at 2024-03-25T19:43:01.438 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 1 librados: starting objecter 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 1 Processor -- start 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 1 -- start start 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 1 librados: setting wanted keys 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 1 librados: calling monclient init 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.435+0000 7f8fe740d640 10 monclient: init 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe00a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe013f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe01430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00b1870 con 0x7f8fe00a98f0 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00ac910 con 0x7f8fe013f710 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00abb00 con 0x7f8fe01430c0 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.436+0000 7f8fe740d640 10 monclient(hunting): authenticate will time out at 4993.705455s 2024-03-25T19:43:01.442 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.443+0000 7f8fe51a9640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8fe00a98f0 0x7f8fe0138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48512/0 (socket says 172.21.15.69:48512) 2024-03-25T19:43:01.442 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.443+0000 7f8fe51a9640 1 -- 172.21.15.69:0/3545941180 learned_addr learned my addr 172.21.15.69:0/3545941180 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:01.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.444+0000 7f8fd5ffb640 10 client.?.objecter ms_handle_connect 0x7f8fe00a98f0 2024-03-25T19:43:01.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.444+0000 7f8fd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:01.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.447+0000 7f8fd5ffb640 10 client.?.objecter ms_handle_connect 0x7f8fe013f710 2024-03-25T19:43:01.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.447+0000 7f8fd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:01.457 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.457+0000 7f8fd5ffb640 10 client.?.objecter ms_handle_connect 0x7f8fe01430c0 2024-03-25T19:43:01.457 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.457+0000 7f8fd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:01.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.461+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411673353 0 0) 0x7f8fe00ac910 con 0x7f8fe013f710 2024-03-25T19:43:01.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.461+0000 7f8fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.461+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.461+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.461+0000 7f8fd5ffb640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:43:01.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.461+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc4004eb0 con 0x7f8fe013f710 2024-03-25T19:43:01.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.462+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431553453 0 0) 0x7f8fe00b2a30 con 0x7f8fe00a98f0 2024-03-25T19:43:01.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.462+0000 7f8fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.462+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.462+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.462+0000 7f8fd5ffb640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:43:01.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.462+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe00ac910 con 0x7f8fe00a98f0 2024-03-25T19:43:01.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.480+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1214851215 0 0) 0x7f8fc4004eb0 con 0x7f8fe013f710 2024-03-25T19:43:01.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.480+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fe00b2a30 con 0x7f8fe013f710 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.480+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8fd0001b00 con 0x7f8fe013f710 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.480+0000 7f8fd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.480+0000 7f8fd5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.480+0000 7f8fd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:01.481 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:01.482 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:01.482 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:01.482 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:01.482 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.480+0000 7f8fd5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:01.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.486+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 252297872 0 0) 0x7f8fe00ac910 con 0x7f8fe00a98f0 2024-03-25T19:43:01.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.486+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fc4004eb0 con 0x7f8fe00a98f0 2024-03-25T19:43:01.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.486+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8fdc001810 con 0x7f8fe00a98f0 2024-03-25T19:43:01.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.486+0000 7f8fd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.486+0000 7f8fd5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:01.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.486+0000 7f8fd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:01.486 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:01.486 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:01.487 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.487 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.487 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.487 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:01.487 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:01.487 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:01.487 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:01.487 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:01.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.487+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620949718 0 0) 0x7f8fe00abb00 con 0x7f8fe01430c0 2024-03-25T19:43:01.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.487+0000 7f8fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.487+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.487+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.487+0000 7f8fd5ffb640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:43:01.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.487+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe00ac910 con 0x7f8fe01430c0 2024-03-25T19:43:01.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3821893965 0 0) 0x7f8fe00b2a30 con 0x7f8fe013f710 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 >> v1:172.21.15.69:6790/0 conn(0x7f8fe01430c0 legacy=0x7f8fe01e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 >> v1:172.21.15.69:6789/0 conn(0x7f8fe00a98f0 legacy=0x7f8fe0138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 1 monclient: found mon.b 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe00abde0 con 0x7f8fe013f710 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:01.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 5 monclient: authenticate success, global_id 4306 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 10 monclient: _renew_subs 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe0138f40 con 0x7f8fe013f710 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 10 monclient: _renew_subs 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe01363f0 con 0x7f8fe013f710 2024-03-25T19:43:01.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.497+0000 7f8fe740d640 1 librados: init done 2024-03-25T19:43:01.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.498+0000 7f8fe740d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:01.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.498+0000 7f8fe740d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:01.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.498+0000 7f8fe740d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:01.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.498+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe00627b0 con 0x7f8fe013f710 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.774+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fd0001220 con 0x7f8fe013f710 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.775+0000 7f8fd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.775+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8fd0004c30 con 0x7f8fe013f710 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.775+0000 7f8fd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.775+0000 7f8fd5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.775+0000 7f8fd5ffb640 10 monclient: dump: 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:01.775 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.776 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:01.776 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.776 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:01.776 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:01.776 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:01.776 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:01.776 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:01.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.775+0000 7f8fd77fe640 4 set_mon_vals no callback set 2024-03-25T19:43:01.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.775+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f8fd001ac00 con 0x7f8fe013f710 2024-03-25T19:43:01.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.776+0000 7f8fe51a9640 10 monclient: get_auth_request con 0x7f8fc406df50 auth_method 0 2024-03-25T19:43:01.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.776+0000 7f8fd5ffb640 10 client.4306.objecter ms_handle_connect 0x7f8fc406df50 2024-03-25T19:43:01.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.778+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f8fd0054890 con 0x7f8fe013f710 2024-03-25T19:43:01.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.778+0000 7f8fd5ffb640 10 client.4306.objecter ms_dispatch 0x7f8fe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:01.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.778+0000 7f8fd5ffb640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:01.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.778+0000 7f8fd5ffb640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:01.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.778+0000 7f8fd5ffb640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T19:43:01.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.781+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fd00503a0 con 0x7f8fe013f710 2024-03-25T19:43:01.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.781+0000 7f8fd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:01.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:01.781+0000 7f8fd5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:01.816 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:39.780 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:01.822 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:39.786 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:01.855 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:39.819 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:01.898 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:39.852 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:01.906 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:39.861 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:01.910 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:39.874 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:01.921 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:39.885 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:01.935 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:39.890 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:01.954 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:39.918 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:02.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.000+0000 7f8fe740d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:02.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.000+0000 7f8fe740d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:02.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.000+0000 7f8fe740d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:02.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.000+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fe01434a0 con 0x7f8fe013f710 2024-03-25T19:43:02.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.054+0000 7f8fd5ffb640 1 -- 172.21.15.69:0/3545941180 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f8fd00522d0 con 0x7f8fe013f710 2024-03-25T19:43:02.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.054+0000 7f8fd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:02.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.054+0000 7f8fd5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:02.054 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:43:02.054 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 10 librados: watch_flush enter 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 10 librados: watch_flush exit 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 10 client.4306.objecter successfully canceled tick 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 >> v1:172.21.15.169:6800/3072548259 conn(0x7f8fc406df50 legacy=0x7f8fc4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 10 monclient: shutdown 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 >> v1:172.21.15.169:6789/0 conn(0x7f8fe013f710 legacy=0x7f8fe00ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 shutdown_connections 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 >> 172.21.15.69:0/3545941180 conn(0x7f8fe00b7ea0 msgr2=0x7f8fe0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 shutdown_connections 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 1 -- 172.21.15.69:0/3545941180 wait complete. 2024-03-25T19:43:02.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:02.056+0000 7f8fe740d640 1 librados: shutdown 2024-03-25T19:43:02.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:08.874 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:08.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.986+0000 7f189f325640 10 monclient: get_monmap_and_config 2024-03-25T19:43:08.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.986+0000 7f189f325640 10 monclient: build_initial_monmap 2024-03-25T19:43:08.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.986+0000 7f189f325640 10 monclient: monmap: 2024-03-25T19:43:08.986 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:08.986 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:08.987 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:43:08.987658+0000 2024-03-25T19:43:08.987 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:43:08.987658+0000 2024-03-25T19:43:08.987 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:08.987 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:08.987 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:08.987 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:43:08.987 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:43:08.987 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:08.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.987+0000 7f189f325640 1 Processor -- start 2024-03-25T19:43:08.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.987+0000 7f189f325640 1 -- start start 2024-03-25T19:43:08.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.987+0000 7f189f325640 10 monclient: init 2024-03-25T19:43:08.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 10 monclient(hunting): picked mon.noname-b con 0x7f18980a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 10 monclient(hunting): picked mon.noname-a con 0x7f189813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 10 monclient(hunting): picked mon.noname-c con 0x7f18981430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898142520 con 0x7f189813f710 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980b28a0 con 0x7f18981430c0 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980b2a30 con 0x7f18980a98f0 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:08.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.988+0000 7f189f325640 10 monclient(hunting): authenticate will time out at 5001.257000s 2024-03-25T19:43:08.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.996+0000 7f189c8c0640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f189813f710 0x7f18981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53286/0 (socket says 172.21.15.69:53286) 2024-03-25T19:43:08.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:08.996+0000 7f189c8c0640 1 -- 172.21.15.69:0/3221468521 learned_addr learned my addr 172.21.15.69:0/3221468521 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:09.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.009+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273006787 0 0) 0x7f18980b2a30 con 0x7f18980a98f0 2024-03-25T19:43:09.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.009+0000 7f188ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.009+0000 7f188ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.009+0000 7f188ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.009+0000 7f188ffff640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:43:09.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.009+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1884003520 con 0x7f18980a98f0 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.015+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249432581 0 0) 0x7f18980b28a0 con 0x7f18981430c0 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.015+0000 7f188ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.015+0000 7f188ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.015+0000 7f188ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.015+0000 7f188ffff640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.015+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18980b2a30 con 0x7f18981430c0 2024-03-25T19:43:09.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2055287054 0 0) 0x7f1884003520 con 0x7f18980a98f0 2024-03-25T19:43:09.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 >> v1:172.21.15.169:6789/0 conn(0x7f18981430c0 legacy=0x7f189814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 >> v1:172.21.15.69:6789/0 conn(0x7f189813f710 legacy=0x7f18981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:09.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 1 monclient: found mon.noname-b 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18980b44c0 con 0x7f18980a98f0 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 10 monclient: _finish_auth 0 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1894001690 con 0x7f18980a98f0 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f188ffff640 10 monclient: dump: 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.023 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.024 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.024 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:09.024 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:09.024 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:09.024 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:09.024 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:09.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f189f325640 5 monclient: authenticate success, global_id 4313 2024-03-25T19:43:09.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.022+0000 7f189f325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:09.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.031+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18940017f0 con 0x7f18980a98f0 2024-03-25T19:43:09.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.031+0000 7f188ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:09.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.031+0000 7f188ffff640 1 -- 172.21.15.69:0/3221468521 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1894001c00 con 0x7f18980a98f0 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.031+0000 7f189f325640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.031+0000 7f189f325640 4 set_mon_vals no callback set 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.031+0000 7f188ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.031+0000 7f188ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.031+0000 7f188ffff640 10 monclient: dump: 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:09.032 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.032+0000 7f189f325640 10 monclient: shutdown 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.032+0000 7f189f325640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.032+0000 7f189f325640 1 -- 172.21.15.69:0/3221468521 >> v1:172.21.15.69:6790/0 conn(0x7f18980a98f0 legacy=0x7f18980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.032+0000 7f189f325640 1 -- 172.21.15.69:0/3221468521 shutdown_connections 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.032+0000 7f189f325640 1 -- 172.21.15.69:0/3221468521 >> 172.21.15.69:0/3221468521 conn(0x7f18980b7ea0 msgr2=0x7f18980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.032+0000 7f189f325640 1 -- 172.21.15.69:0/3221468521 shutdown_connections 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.032+0000 7f189f325640 1 -- 172.21.15.69:0/3221468521 wait complete. 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.033+0000 7f189f325640 10 monclient: build_initial_monmap 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.033+0000 7f189f325640 10 monclient: monmap: 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.033+0000 7f189f325640 1 librados: starting msgr at 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.033+0000 7f189f325640 1 librados: starting objecter 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.033+0000 7f189f325640 1 Processor -- start 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 1 -- start start 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 1 librados: setting wanted keys 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 1 librados: calling monclient init 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 10 monclient: init 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 10 monclient(hunting): picked mon.noname-c con 0x7f18980a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 10 monclient(hunting): picked mon.noname-a con 0x7f189813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 10 monclient(hunting): picked mon.noname-b con 0x7f18981430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898138b10 con 0x7f189813f710 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898136e10 con 0x7f18981430c0 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898138d60 con 0x7f18980a98f0 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.034+0000 7f189f325640 10 monclient(hunting): authenticate will time out at 5001.303593s 2024-03-25T19:43:09.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.043+0000 7f189d0c1640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f18980a98f0 0x7f1898137560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:56264/0 (socket says 172.21.15.69:56264) 2024-03-25T19:43:09.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.043+0000 7f189d0c1640 1 -- 172.21.15.69:0/944256335 learned_addr learned my addr 172.21.15.69:0/944256335 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:09.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.045+0000 7f188dffb640 10 client.?.objecter ms_handle_connect 0x7f18980a98f0 2024-03-25T19:43:09.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.045+0000 7f188dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:09.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.048+0000 7f188dffb640 10 client.?.objecter ms_handle_connect 0x7f18981430c0 2024-03-25T19:43:09.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.048+0000 7f188dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:09.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.057+0000 7f188dffb640 10 client.?.objecter ms_handle_connect 0x7f189813f710 2024-03-25T19:43:09.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.057+0000 7f188dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:09.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.061+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902407125 0 0) 0x7f1898136e10 con 0x7f18981430c0 2024-03-25T19:43:09.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.061+0000 7f188dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.061+0000 7f188dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.062 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.061+0000 7f188dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.062 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.061+0000 7f188dffb640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:43:09.062 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.061+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f187c004eb0 con 0x7f18981430c0 2024-03-25T19:43:09.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.067+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32043326 0 0) 0x7f1898138d60 con 0x7f18980a98f0 2024-03-25T19:43:09.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.067+0000 7f188dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.067+0000 7f188dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.067+0000 7f188dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.067+0000 7f188dffb640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:43:09.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.067+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898136e10 con 0x7f18980a98f0 2024-03-25T19:43:09.069 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.069+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490668958 0 0) 0x7f1898138b10 con 0x7f189813f710 2024-03-25T19:43:09.069 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.069+0000 7f188dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.069 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.069+0000 7f188dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.069 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.069+0000 7f188dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.069 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.069+0000 7f188dffb640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:43:09.069 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.069+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898138d60 con 0x7f189813f710 2024-03-25T19:43:09.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.079+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 646596401 0 0) 0x7f1898136e10 con 0x7f18980a98f0 2024-03-25T19:43:09.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.079+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1898138b10 con 0x7f18980a98f0 2024-03-25T19:43:09.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.079+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1894001690 con 0x7f18980a98f0 2024-03-25T19:43:09.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.079+0000 7f188dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.079+0000 7f188dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:09.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.079+0000 7f188dffb640 10 monclient(hunting): dump: 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:09.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.079+0000 7f188dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.082+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1104446851 0 0) 0x7f1898138d60 con 0x7f189813f710 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.082+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1898136e10 con 0x7f189813f710 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.082+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1888001860 con 0x7f189813f710 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.082+0000 7f188dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.082+0000 7f188dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.082+0000 7f188dffb640 10 monclient(hunting): dump: 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.083 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.083+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 116231948 0 0) 0x7f187c004eb0 con 0x7f18981430c0 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.083+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1898138d60 con 0x7f18981430c0 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.083+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1890001bd0 con 0x7f18981430c0 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.083+0000 7f188dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.083+0000 7f188dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.083+0000 7f188dffb640 10 monclient(hunting): dump: 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.084 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.085 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:09.085 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:09.085 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:09.085 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:09.085 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:09.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1108433849 0 0) 0x7f1898138b10 con 0x7f18980a98f0 2024-03-25T19:43:09.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 >> v1:172.21.15.169:6789/0 conn(0x7f18981430c0 legacy=0x7f18981e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 >> v1:172.21.15.69:6789/0 conn(0x7f189813f710 legacy=0x7f18981df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 1 monclient: found mon.c 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1898139040 con 0x7f18980a98f0 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f188dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 5 monclient: authenticate success, global_id 4316 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 10 monclient: _renew_subs 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1898139380 con 0x7f18980a98f0 2024-03-25T19:43:09.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:09.088 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 10 monclient: _renew_subs 2024-03-25T19:43:09.088 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:09.088 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1898139920 con 0x7f18980a98f0 2024-03-25T19:43:09.088 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.086+0000 7f189f325640 1 librados: init done 2024-03-25T19:43:09.088 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.087+0000 7f189f325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.087+0000 7f189f325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.087+0000 7f189f325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.087+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1898074cc0 con 0x7f18980a98f0 2024-03-25T19:43:09.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.389+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18940038a0 con 0x7f18980a98f0 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.389+0000 7f188dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.389+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1894003bf0 con 0x7f18980a98f0 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.389+0000 7f188dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.389+0000 7f188dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.389+0000 7f188dffb640 10 monclient: dump: 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:09.390 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:09.391 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:09.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.389+0000 7f188f7fe640 4 set_mon_vals no callback set 2024-03-25T19:43:09.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.390+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f1894003e10 con 0x7f18980a98f0 2024-03-25T19:43:09.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.391+0000 7f189c8c0640 10 monclient: get_auth_request con 0x7f187c06e0f0 auth_method 0 2024-03-25T19:43:09.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.391+0000 7f188dffb640 10 client.4316.objecter ms_handle_connect 0x7f187c06e0f0 2024-03-25T19:43:09.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.396+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f1894054880 con 0x7f18980a98f0 2024-03-25T19:43:09.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.396+0000 7f188dffb640 10 client.4316.objecter ms_dispatch 0x7f1898000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:09.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.396+0000 7f188dffb640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:09.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.396+0000 7f188dffb640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:09.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.396+0000 7f188dffb640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-25T19:43:09.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.396+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1894050390 con 0x7f18980a98f0 2024-03-25T19:43:09.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.396+0000 7f188dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:09.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.396+0000 7f188dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:09.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.645+0000 7f189f325640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:09.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.645+0000 7f189f325640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:09.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.645+0000 7f189f325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:09.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.645+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18981391d0 con 0x7f18980a98f0 2024-03-25T19:43:09.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.699+0000 7f188dffb640 1 -- 172.21.15.69:0/944256335 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f18940522c0 con 0x7f18980a98f0 2024-03-25T19:43:09.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.699+0000 7f188dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:09.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.699+0000 7f188dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:09.699 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:43:09.699 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 10 librados: watch_flush enter 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 10 librados: watch_flush exit 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 10 client.4316.objecter successfully canceled tick 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 >> v1:172.21.15.169:6800/3072548259 conn(0x7f187c06e0f0 legacy=0x7f187c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 10 monclient: shutdown 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 >> v1:172.21.15.69:6790/0 conn(0x7f18980a98f0 legacy=0x7f1898137560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 shutdown_connections 2024-03-25T19:43:09.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 >> 172.21.15.69:0/944256335 conn(0x7f18980b7ea0 msgr2=0x7f1898074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:09.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 shutdown_connections 2024-03-25T19:43:09.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 1 -- 172.21.15.69:0/944256335 wait complete. 2024-03-25T19:43:09.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:09.701+0000 7f189f325640 1 librados: shutdown 2024-03-25T19:43:09.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:16.519 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:16.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.630+0000 7fcb619f4640 10 monclient: get_monmap_and_config 2024-03-25T19:43:16.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.630+0000 7fcb619f4640 10 monclient: build_initial_monmap 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.630+0000 7fcb619f4640 10 monclient: monmap: 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:43:16.631988+0000 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:43:16.631988+0000 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:43:16.631 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 1 Processor -- start 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 1 -- start start 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 10 monclient: init 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 10 monclient(hunting): picked mon.noname-b con 0x7fcb5c0b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 10 monclient(hunting): picked mon.noname-c con 0x7fcb5c0b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 10 monclient(hunting): picked mon.noname-a con 0x7fcb5c14be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c0af940 con 0x7fcb5c14be90 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.631+0000 7fcb619f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c0b0a70 con 0x7fcb5c0b1730 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.632+0000 7fcb619f4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c0b0d50 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.632+0000 7fcb619f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:16.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.632+0000 7fcb619f4640 10 monclient(hunting): authenticate will time out at 5008.900627s 2024-03-25T19:43:16.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.632+0000 7fcb5affd640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fcb5c0b4d60 0x7fcb5c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:43520/0 (socket says 172.21.15.69:43520) 2024-03-25T19:43:16.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.632+0000 7fcb5affd640 1 -- 172.21.15.69:0/1806401601 learned_addr learned my addr 172.21.15.69:0/1806401601 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:16.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.705+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357803842 0 0) 0x7fcb5c0af940 con 0x7fcb5c14be90 2024-03-25T19:43:16.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.705+0000 7fcb5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.705+0000 7fcb5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.705+0000 7fcb5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.705+0000 7fcb5a7fc640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:43:16.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.705+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb48003520 con 0x7fcb5c14be90 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1546495772 0 0) 0x7fcb48003520 con 0x7fcb5c14be90 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 >> v1:172.21.15.69:6790/0 conn(0x7fcb5c0b4d60 legacy=0x7fcb5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 >> v1:172.21.15.169:6789/0 conn(0x7fcb5c0b1730 legacy=0x7fcb5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c0b1030 con 0x7fcb5c14be90 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:16.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcb54001a10 con 0x7fcb5c14be90 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.725+0000 7fcb5a7fc640 10 monclient: dump: 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:16.727 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:16.728 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:16.728 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:16.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.726+0000 7fcb619f4640 5 monclient: authenticate success, global_id 4341 2024-03-25T19:43:16.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.726+0000 7fcb619f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:16.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.733+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb54001b70 con 0x7fcb5c14be90 2024-03-25T19:43:16.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.733+0000 7fcb5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:16.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.733+0000 7fcb5a7fc640 1 -- 172.21.15.69:0/1806401601 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcb54001f80 con 0x7fcb5c14be90 2024-03-25T19:43:16.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.733+0000 7fcb619f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:16.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.733+0000 7fcb619f4640 4 set_mon_vals no callback set 2024-03-25T19:43:16.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.733+0000 7fcb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.733 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.733+0000 7fcb5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.733+0000 7fcb5a7fc640 10 monclient: dump: 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:16.734 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:16.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.734+0000 7fcb619f4640 10 monclient: shutdown 2024-03-25T19:43:16.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.734+0000 7fcb619f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.734+0000 7fcb619f4640 1 -- 172.21.15.69:0/1806401601 >> v1:172.21.15.69:6789/0 conn(0x7fcb5c14be90 legacy=0x7fcb5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.734+0000 7fcb619f4640 1 -- 172.21.15.69:0/1806401601 shutdown_connections 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.734+0000 7fcb619f4640 1 -- 172.21.15.69:0/1806401601 >> 172.21.15.69:0/1806401601 conn(0x7fcb5c0617a0 msgr2=0x7fcb5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.734+0000 7fcb619f4640 1 -- 172.21.15.69:0/1806401601 shutdown_connections 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.734+0000 7fcb619f4640 1 -- 172.21.15.69:0/1806401601 wait complete. 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 10 monclient: build_initial_monmap 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 10 monclient: monmap: 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:16.737 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 1 librados: starting msgr at 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 1 librados: starting objecter 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 1 Processor -- start 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 1 -- start start 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 1 librados: setting wanted keys 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 1 librados: calling monclient init 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.735+0000 7fcb619f4640 10 monclient: init 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 10 monclient(hunting): picked mon.noname-c con 0x7fcb5c0b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:16.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 10 monclient(hunting): picked mon.noname-a con 0x7fcb5c0b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:16.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 10 monclient(hunting): picked mon.noname-b con 0x7fcb5c14be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:16.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c137760 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c1387d0 con 0x7fcb5c14be90 2024-03-25T19:43:16.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c140380 con 0x7fcb5c0b1730 2024-03-25T19:43:16.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:16.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.736+0000 7fcb619f4640 10 monclient(hunting): authenticate will time out at 5009.005148s 2024-03-25T19:43:16.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.751+0000 7fcb5b7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fcb5c14be90 0x7fcb5c1e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37120/0 (socket says 172.21.15.69:37120) 2024-03-25T19:43:16.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.751+0000 7fcb5b7fe640 1 -- 172.21.15.69:0/3109538298 learned_addr learned my addr 172.21.15.69:0/3109538298 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:16.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.751+0000 7fcb58ff9640 10 client.?.objecter ms_handle_connect 0x7fcb5c14be90 2024-03-25T19:43:16.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.751+0000 7fcb58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:16.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.755+0000 7fcb58ff9640 10 client.?.objecter ms_handle_connect 0x7fcb5c0b1730 2024-03-25T19:43:16.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.755+0000 7fcb58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:16.809 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.809+0000 7fcb58ff9640 10 client.?.objecter ms_handle_connect 0x7fcb5c0b4d60 2024-03-25T19:43:16.809 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.809+0000 7fcb58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:16.816 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.816+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478754800 0 0) 0x7fcb5c1387d0 con 0x7fcb5c14be90 2024-03-25T19:43:16.816 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.816+0000 7fcb58ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.816 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.816+0000 7fcb58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.817 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.816+0000 7fcb58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.817 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.816+0000 7fcb58ff9640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:43:16.817 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.816+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb3c004eb0 con 0x7fcb5c14be90 2024-03-25T19:43:16.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.821+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510168351 0 0) 0x7fcb5c137760 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.821+0000 7fcb58ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.821+0000 7fcb58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.821+0000 7fcb58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.821+0000 7fcb58ff9640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:43:16.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.821+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb5c1387d0 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.836+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641966018 0 0) 0x7fcb5c1387d0 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.836+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb5c137760 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.836+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcb440018a0 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.836+0000 7fcb58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.836+0000 7fcb58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.836+0000 7fcb58ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:16.837 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.836+0000 7fcb58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:16.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.842+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3785221246 0 0) 0x7fcb3c004eb0 con 0x7fcb5c14be90 2024-03-25T19:43:16.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.842+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb5c1387d0 con 0x7fcb5c14be90 2024-03-25T19:43:16.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.842+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcb54001fc0 con 0x7fcb5c14be90 2024-03-25T19:43:16.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.842+0000 7fcb58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.842+0000 7fcb58ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:16.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.842+0000 7fcb58ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:16.842 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:16.843 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.847+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1591396024 0 0) 0x7fcb5c137760 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.847+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 >> v1:172.21.15.69:6790/0 conn(0x7fcb5c0b1730 legacy=0x7fcb5c140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.847+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 >> v1:172.21.15.169:6789/0 conn(0x7fcb5c14be90 legacy=0x7fcb5c1e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.847+0000 7fcb58ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.847+0000 7fcb58ff9640 1 monclient: found mon.a 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.847+0000 7fcb58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb58ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c136910 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb58ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:16.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb58ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 5 monclient: authenticate success, global_id 4344 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 10 monclient: _renew_subs 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c136b60 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 10 monclient: _renew_subs 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5c1372b0 con 0x7fcb5c0b4d60 2024-03-25T19:43:16.849 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.848+0000 7fcb619f4640 1 librados: init done 2024-03-25T19:43:16.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.849+0000 7fcb619f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:16.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.849+0000 7fcb619f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:16.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.849+0000 7fcb619f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:43:16.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:16.849+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5c140510 con 0x7fcb5c0b4d60 2024-03-25T19:43:17.199 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.199+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb44001d50 con 0x7fcb5c0b4d60 2024-03-25T19:43:17.199 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.199+0000 7fcb58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:17.199 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.199+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcb440037d0 con 0x7fcb5c0b4d60 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.199+0000 7fcb58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.199+0000 7fcb58ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.199+0000 7fcb58ff9640 10 monclient: dump: 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:17.200 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:17.201 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:17.201 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:17.201 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:17.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.199+0000 7fcb59ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:17.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.199+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fcb440039f0 con 0x7fcb5c0b4d60 2024-03-25T19:43:17.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.200+0000 7fcb5affd640 10 monclient: get_auth_request con 0x7fcb3c06de40 auth_method 0 2024-03-25T19:43:17.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.200+0000 7fcb58ff9640 10 client.4344.objecter ms_handle_connect 0x7fcb3c06de40 2024-03-25T19:43:17.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.205+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fcb44053570 con 0x7fcb5c0b4d60 2024-03-25T19:43:17.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.205+0000 7fcb58ff9640 10 client.4344.objecter ms_dispatch 0x7fcb5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:17.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.205+0000 7fcb58ff9640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:17.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.205+0000 7fcb58ff9640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:17.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.205+0000 7fcb58ff9640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-25T19:43:17.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.205+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb4404eeb0 con 0x7fcb5c0b4d60 2024-03-25T19:43:17.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.205+0000 7fcb58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:17.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.205+0000 7fcb58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:17.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.444+0000 7fcb619f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:17.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.444+0000 7fcb619f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:17.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.444+0000 7fcb619f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:43:17.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.444+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb5c137550 con 0x7fcb5c0b4d60 2024-03-25T19:43:17.473 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.473+0000 7fcb58ff9640 1 -- 172.21.15.69:0/3109538298 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fcb44050fb0 con 0x7fcb5c0b4d60 2024-03-25T19:43:17.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.473+0000 7fcb58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:17.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.473+0000 7fcb58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:17.474 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:43:17.474 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:17.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 10 librados: watch_flush enter 2024-03-25T19:43:17.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 10 librados: watch_flush exit 2024-03-25T19:43:17.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 10 client.4344.objecter successfully canceled tick 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 >> v1:172.21.15.169:6800/3072548259 conn(0x7fcb3c06de40 legacy=0x7fcb3c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 10 monclient: shutdown 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 >> v1:172.21.15.69:6789/0 conn(0x7fcb5c0b4d60 legacy=0x7fcb5c144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 shutdown_connections 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 >> 172.21.15.69:0/3109538298 conn(0x7fcb5c0617a0 msgr2=0x7fcb5c0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 shutdown_connections 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 1 -- 172.21.15.69:0/3109538298 wait complete. 2024-03-25T19:43:17.477 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:17.476+0000 7fcb619f4640 1 librados: shutdown 2024-03-25T19:43:17.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:21.614 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:59.578 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.670 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:59.634 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.702 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:59.656 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.715 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:59.679 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.716 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:59.670 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.731 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:59.696 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.735 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:59.699 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.744 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:59.698 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.755 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:59.719 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:24.293 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.407+0000 7f35df182640 10 monclient: get_monmap_and_config 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.407+0000 7f35df182640 10 monclient: build_initial_monmap 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.407+0000 7f35df182640 10 monclient: monmap: 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:43:24.408420+0000 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:43:24.408420+0000 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:24.407 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:24.408 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:24.408 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:43:24.408 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:43:24.408 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.408+0000 7f35df182640 1 Processor -- start 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.408+0000 7f35df182640 1 -- start start 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.408+0000 7f35df182640 10 monclient: init 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 10 monclient(hunting): picked mon.noname-b con 0x7f35d80a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 10 monclient(hunting): picked mon.noname-c con 0x7f35d813f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 10 monclient(hunting): picked mon.noname-a con 0x7f35d81430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d8142520 con 0x7f35d81430c0 2024-03-25T19:43:24.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d80b28a0 con 0x7f35d813f710 2024-03-25T19:43:24.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d80b2a30 con 0x7f35d80a98f0 2024-03-25T19:43:24.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:24.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.409+0000 7f35df182640 10 monclient(hunting): authenticate will time out at 5016.677801s 2024-03-25T19:43:24.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.410+0000 7f35d7fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f35d813f710 0x7f35d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:35752/0 (socket says 172.21.15.69:35752) 2024-03-25T19:43:24.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.410+0000 7f35d7fff640 1 -- 172.21.15.69:0/978251202 learned_addr learned my addr 172.21.15.69:0/978251202 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:24.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.498+0000 7f35d77fe640 1 -- 172.21.15.69:0/978251202 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609953852 0 0) 0x7f35d8142520 con 0x7f35d81430c0 2024-03-25T19:43:24.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.498+0000 7f35d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.498+0000 7f35d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.498+0000 7f35d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.498+0000 7f35d77fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:43:24.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.498+0000 7f35d77fe640 1 -- 172.21.15.69:0/978251202 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c8003520 con 0x7f35d81430c0 2024-03-25T19:43:24.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 1 -- 172.21.15.69:0/978251202 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3000990481 0 0) 0x7f35c8003520 con 0x7f35d81430c0 2024-03-25T19:43:24.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 1 -- 172.21.15.69:0/978251202 >> v1:172.21.15.69:6790/0 conn(0x7f35d80a98f0 legacy=0x7f35d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 1 -- 172.21.15.69:0/978251202 >> v1:172.21.15.169:6789/0 conn(0x7f35d813f710 legacy=0x7f35d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:24.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 1 monclient: found mon.noname-a 2024-03-25T19:43:24.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:24.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:43:24.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 1 -- 172.21.15.69:0/978251202 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d80b44c0 con 0x7f35d81430c0 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 1 -- 172.21.15.69:0/978251202 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f35d0001a10 con 0x7f35d81430c0 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35df182640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35df182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35d77fe640 10 monclient: dump: 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:24.510 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.511 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:24.511 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:24.511 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:24.511 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:24.511 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:24.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.510+0000 7f35df182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:24.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.516+0000 7f35d77fe640 1 -- 172.21.15.69:0/978251202 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35d0001b70 con 0x7f35d81430c0 2024-03-25T19:43:24.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.516+0000 7f35d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:24.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.516+0000 7f35df182640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:24.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.516+0000 7f35df182640 4 set_mon_vals no callback set 2024-03-25T19:43:24.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.517+0000 7f35df182640 10 monclient: shutdown 2024-03-25T19:43:24.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.517+0000 7f35df182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:24.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.517+0000 7f35df182640 1 -- 172.21.15.69:0/978251202 >> v1:172.21.15.69:6789/0 conn(0x7f35d81430c0 legacy=0x7f35d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.517+0000 7f35df182640 1 -- 172.21.15.69:0/978251202 shutdown_connections 2024-03-25T19:43:24.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.517+0000 7f35df182640 1 -- 172.21.15.69:0/978251202 >> 172.21.15.69:0/978251202 conn(0x7f35d80b7ea0 msgr2=0x7f35d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:24.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.517+0000 7f35df182640 1 -- 172.21.15.69:0/978251202 shutdown_connections 2024-03-25T19:43:24.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.517+0000 7f35df182640 1 -- 172.21.15.69:0/978251202 wait complete. 2024-03-25T19:43:24.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.518+0000 7f35df182640 10 monclient: build_initial_monmap 2024-03-25T19:43:24.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.518+0000 7f35df182640 10 monclient: monmap: 2024-03-25T19:43:24.518 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:24.518 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:24.518 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:43:24.518 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:43:24.518 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:24.518 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.519+0000 7f35df182640 1 librados: starting msgr at 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.519+0000 7f35df182640 1 librados: starting objecter 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.519+0000 7f35df182640 1 Processor -- start 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.519+0000 7f35df182640 1 -- start start 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.519+0000 7f35df182640 1 librados: setting wanted keys 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.519+0000 7f35df182640 1 librados: calling monclient init 2024-03-25T19:43:24.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.519+0000 7f35df182640 10 monclient: init 2024-03-25T19:43:24.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:24.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:24.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 10 monclient(hunting): picked mon.noname-b con 0x7f35d80a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:24.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 10 monclient(hunting): picked mon.noname-a con 0x7f35d813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:24.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 10 monclient(hunting): picked mon.noname-c con 0x7f35d81430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:24.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d81d77a0 con 0x7f35d813f710 2024-03-25T19:43:24.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d81d8340 con 0x7f35d80a98f0 2024-03-25T19:43:24.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d80abb90 con 0x7f35d81430c0 2024-03-25T19:43:24.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:24.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.520+0000 7f35df182640 10 monclient(hunting): authenticate will time out at 5016.788136s 2024-03-25T19:43:24.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.524+0000 7f35d7fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f35d813f710 0x7f35d81d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49174/0 (socket says 172.21.15.69:49174) 2024-03-25T19:43:24.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.524+0000 7f35d7fff640 1 -- 172.21.15.69:0/3273268241 learned_addr learned my addr 172.21.15.69:0/3273268241 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:24.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.527+0000 7f35d57fa640 10 client.?.objecter ms_handle_connect 0x7f35d813f710 2024-03-25T19:43:24.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.527+0000 7f35d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:24.537 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.539+0000 7f35d57fa640 10 client.?.objecter ms_handle_connect 0x7f35d81430c0 2024-03-25T19:43:24.537 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.539+0000 7f35d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:24.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.544+0000 7f35d57fa640 10 client.?.objecter ms_handle_connect 0x7f35d80a98f0 2024-03-25T19:43:24.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.544+0000 7f35d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:24.543 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.544+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560613179 0 0) 0x7f35d80abb90 con 0x7f35d81430c0 2024-03-25T19:43:24.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:43:24.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc004eb0 con 0x7f35d81430c0 2024-03-25T19:43:24.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227742850 0 0) 0x7f35d81d77a0 con 0x7f35d813f710 2024-03-25T19:43:24.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.545 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.545 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.545+0000 7f35d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.545 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.546+0000 7f35d57fa640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:43:24.545 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.546+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d80abb90 con 0x7f35d813f710 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.559+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 753041331 0 0) 0x7f35bc004eb0 con 0x7f35d81430c0 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.559+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35d81d77a0 con 0x7f35d81430c0 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.559+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f35d0001f40 con 0x7f35d81430c0 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.559+0000 7f35d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.559+0000 7f35d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.559+0000 7f35d57fa640 10 monclient(hunting): dump: 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:24.558 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:24.559 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.559 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:24.559 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:24.559 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:24.559 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:24.559 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:24.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.559+0000 7f35d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:24.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.566+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640494524 0 0) 0x7f35d81d8340 con 0x7f35d80a98f0 2024-03-25T19:43:24.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.566+0000 7f35d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.566+0000 7f35d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.566+0000 7f35d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.566+0000 7f35d57fa640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:43:24.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.566+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc004eb0 con 0x7f35d80a98f0 2024-03-25T19:43:24.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.567+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4246211073 0 0) 0x7f35d80abb90 con 0x7f35d813f710 2024-03-25T19:43:24.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.567+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35d81d8340 con 0x7f35d813f710 2024-03-25T19:43:24.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.567+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f35c40018f0 con 0x7f35d813f710 2024-03-25T19:43:24.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.567+0000 7f35d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.567+0000 7f35d57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:24.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.567+0000 7f35d57fa640 10 monclient(hunting): dump: 2024-03-25T19:43:24.566 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:24.567 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2676060177 0 0) 0x7f35d81d77a0 con 0x7f35d81430c0 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 >> v1:172.21.15.169:6789/0 conn(0x7f35d80a98f0 legacy=0x7f35d8138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 >> v1:172.21.15.69:6789/0 conn(0x7f35d813f710 legacy=0x7f35d81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 1 monclient: found mon.c 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d81378a0 con 0x7f35d81430c0 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:24.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 5 monclient: authenticate success, global_id 4328 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 10 monclient: _renew_subs 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d8137af0 con 0x7f35d81430c0 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 10 monclient: _renew_subs 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d8135260 con 0x7f35d81430c0 2024-03-25T19:43:24.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.571+0000 7f35df182640 1 librados: init done 2024-03-25T19:43:24.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.572+0000 7f35df182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:24.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.572+0000 7f35df182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:24.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.572+0000 7f35df182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:24.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.572+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d81d7500 con 0x7f35d81430c0 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.864+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35d0003c20 con 0x7f35d81430c0 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.864+0000 7f35d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.864+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f35d00015d0 con 0x7f35d81430c0 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.864+0000 7f35d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.864+0000 7f35d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.864+0000 7f35d57fa640 10 monclient: dump: 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:24.864 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:24.865 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:24.865 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.865 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:24.865 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:24.865 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:24.865 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:24.865 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:24.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.864+0000 7f35d6ffd640 4 set_mon_vals no callback set 2024-03-25T19:43:24.866 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.865+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f35d0019da0 con 0x7f35d81430c0 2024-03-25T19:43:24.866 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.866+0000 7f35dcf1e640 10 monclient: get_auth_request con 0x7f35bc06dfb0 auth_method 0 2024-03-25T19:43:24.866 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.866+0000 7f35d57fa640 10 client.4328.objecter ms_handle_connect 0x7f35bc06dfb0 2024-03-25T19:43:24.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.872+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f35d00539f0 con 0x7f35d81430c0 2024-03-25T19:43:24.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.872+0000 7f35d57fa640 10 client.4328.objecter ms_dispatch 0x7f35d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:24.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.872+0000 7f35d57fa640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:24.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.872+0000 7f35d57fa640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:24.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.872+0000 7f35d57fa640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-25T19:43:24.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.872+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35d004f330 con 0x7f35d81430c0 2024-03-25T19:43:24.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.872+0000 7f35d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:24.871 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:24.872+0000 7f35d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:25.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.112+0000 7f35df182640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:25.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.112+0000 7f35df182640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:25.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.112+0000 7f35df182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:25.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.112+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35d81d7950 con 0x7f35d81430c0 2024-03-25T19:43:25.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.128+0000 7f35d57fa640 1 -- 172.21.15.69:0/3273268241 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f35d81d7950 con 0x7f35d81430c0 2024-03-25T19:43:25.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.129+0000 7f35d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:25.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.129+0000 7f35d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:25.128 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:43:25.128 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 10 librados: watch_flush enter 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 10 librados: watch_flush exit 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 10 client.4328.objecter successfully canceled tick 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 >> v1:172.21.15.169:6800/3072548259 conn(0x7f35bc06dfb0 legacy=0x7f35bc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 10 monclient: shutdown 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 >> v1:172.21.15.69:6790/0 conn(0x7f35d81430c0 legacy=0x7f35d81e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:25.131 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 shutdown_connections 2024-03-25T19:43:25.132 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 >> 172.21.15.69:0/3273268241 conn(0x7f35d80b7ea0 msgr2=0x7f35d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:25.132 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 shutdown_connections 2024-03-25T19:43:25.132 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.131+0000 7f35df182640 1 -- 172.21.15.69:0/3273268241 wait complete. 2024-03-25T19:43:25.132 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:25.132+0000 7f35df182640 1 librados: shutdown 2024-03-25T19:43:25.142 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:31.948 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.065+0000 7f0e6390e640 10 monclient: get_monmap_and_config 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.065+0000 7f0e6390e640 10 monclient: build_initial_monmap 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.065+0000 7f0e6390e640 10 monclient: monmap: 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:43:32.067166+0000 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:43:32.067166+0000 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:32.066 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:43:32.067 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:43:32.067 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.066+0000 7f0e6390e640 1 Processor -- start 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 1 -- start start 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 10 monclient: init 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 10 monclient(hunting): picked mon.noname-a con 0x7f0e5c06c380 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 10 monclient(hunting): picked mon.noname-b con 0x7f0e5c0b12d0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e5c0aedf0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0b16f0 con 0x7f0e5c06c380 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0adfe0 con 0x7f0e5c0aedf0 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0add00 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:32.069 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.067+0000 7f0e6390e640 10 monclient(hunting): authenticate will time out at 5024.336543s 2024-03-25T19:43:32.071 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.071+0000 7f0e60ea9640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f0e5c0b12d0 0x7f0e5c0ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:38044/0 (socket says 172.21.15.69:38044) 2024-03-25T19:43:32.071 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.071+0000 7f0e60ea9640 1 -- 172.21.15.69:0/1400302553 learned_addr learned my addr 172.21.15.69:0/1400302553 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:32.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.086+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440690652 0 0) 0x7f0e5c0add00 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.086+0000 7f0e53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:32.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.086+0000 7f0e53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:32.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.086+0000 7f0e53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:32.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.086+0000 7f0e53fff640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:43:32.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.086+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e48003520 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.094+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145534341 0 0) 0x7f0e5c0b16f0 con 0x7f0e5c06c380 2024-03-25T19:43:32.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.094+0000 7f0e53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:32.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.094+0000 7f0e53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:32.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.094+0000 7f0e53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:32.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.094+0000 7f0e53fff640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:43:32.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.094+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c0add00 con 0x7f0e5c06c380 2024-03-25T19:43:32.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178617548 0 0) 0x7f0e48003520 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 >> v1:172.21.15.169:6789/0 conn(0x7f0e5c0aedf0 legacy=0x7f0e5c14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 >> v1:172.21.15.69:6789/0 conn(0x7f0e5c06c380 legacy=0x7f0e5c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 1 monclient: found mon.noname-b 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c0ae2c0 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0e4c001690 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e6390e640 5 monclient: authenticate success, global_id 4331 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e6390e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:32.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e53fff640 10 monclient: dump: 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:32.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.097+0000 7f0e6390e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:32.104 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.104+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e4c0018b0 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.104 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.104+0000 7f0e53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:32.104 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.104+0000 7f0e53fff640 1 -- 172.21.15.69:0/1400302553 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0e4c001bc0 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.104 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.104+0000 7f0e6390e640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:32.104 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.104+0000 7f0e6390e640 4 set_mon_vals no callback set 2024-03-25T19:43:32.104 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.104+0000 7f0e53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.104+0000 7f0e53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.104+0000 7f0e53fff640 10 monclient: dump: 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:32.105 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:32.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.105+0000 7f0e6390e640 10 monclient: shutdown 2024-03-25T19:43:32.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.105+0000 7f0e6390e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:32.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.105+0000 7f0e6390e640 1 -- 172.21.15.69:0/1400302553 >> v1:172.21.15.69:6790/0 conn(0x7f0e5c0b12d0 legacy=0x7f0e5c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.105+0000 7f0e6390e640 1 -- 172.21.15.69:0/1400302553 shutdown_connections 2024-03-25T19:43:32.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.105+0000 7f0e6390e640 1 -- 172.21.15.69:0/1400302553 >> 172.21.15.69:0/1400302553 conn(0x7f0e5c0b5d10 msgr2=0x7f0e5c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:32.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.105+0000 7f0e6390e640 1 -- 172.21.15.69:0/1400302553 shutdown_connections 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.105+0000 7f0e6390e640 1 -- 172.21.15.69:0/1400302553 wait complete. 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 10 monclient: build_initial_monmap 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 10 monclient: monmap: 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 1 librados: starting msgr at 2024-03-25T19:43:32.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 1 librados: starting objecter 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 1 Processor -- start 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 1 -- start start 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 1 librados: setting wanted keys 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 1 librados: calling monclient init 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.106+0000 7f0e6390e640 10 monclient: init 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e5c06c380 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 10 monclient(hunting): picked mon.noname-a con 0x7f0e5c0aedf0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 10 monclient(hunting): picked mon.noname-b con 0x7f0e5c0b12d0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c06c8e0 con 0x7f0e5c0aedf0 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c1c4750 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c1c3680 con 0x7f0e5c06c380 2024-03-25T19:43:32.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:32.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.107+0000 7f0e6390e640 10 monclient(hunting): authenticate will time out at 5024.376193s 2024-03-25T19:43:32.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.119+0000 7f0e60ea9640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0e5c0aedf0 0x7f0e5c1c0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43112/0 (socket says 172.21.15.69:43112) 2024-03-25T19:43:32.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.119+0000 7f0e60ea9640 1 -- 172.21.15.69:0/3094213285 learned_addr learned my addr 172.21.15.69:0/3094213285 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:32.120 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.120+0000 7f0e51ffb640 10 client.?.objecter ms_handle_connect 0x7f0e5c0aedf0 2024-03-25T19:43:32.120 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.120+0000 7f0e51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:32.120 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.120+0000 7f0e51ffb640 10 client.?.objecter ms_handle_connect 0x7f0e5c06c380 2024-03-25T19:43:32.120 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.120+0000 7f0e51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:32.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.125+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731078742 0 0) 0x7f0e5c06c8e0 con 0x7f0e5c0aedf0 2024-03-25T19:43:32.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.125+0000 7f0e51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:32.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.125+0000 7f0e51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:32.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.125+0000 7f0e51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:32.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.125+0000 7f0e51ffb640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:43:32.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.125+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e40004eb0 con 0x7f0e5c0aedf0 2024-03-25T19:43:32.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.128+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205990327 0 0) 0x7f0e5c1c3680 con 0x7f0e5c06c380 2024-03-25T19:43:32.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.128+0000 7f0e51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:32.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.128+0000 7f0e51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:32.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.128+0000 7f0e51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:32.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.128+0000 7f0e51ffb640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:43:32.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.128+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c06c8e0 con 0x7f0e5c06c380 2024-03-25T19:43:32.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.130+0000 7f0e51ffb640 10 client.?.objecter ms_handle_connect 0x7f0e5c0b12d0 2024-03-25T19:43:32.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.130+0000 7f0e51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:32.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.140+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1206696008 0 0) 0x7f0e5c06c8e0 con 0x7f0e5c06c380 2024-03-25T19:43:32.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.140+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e5c1c3680 con 0x7f0e5c06c380 2024-03-25T19:43:32.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.140+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0e58001930 con 0x7f0e5c06c380 2024-03-25T19:43:32.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.140+0000 7f0e51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:32.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.140+0000 7f0e51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:32.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.140+0000 7f0e51ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:32.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.140+0000 7f0e51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:32.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.142+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383754818 0 0) 0x7f0e5c1c4750 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.142+0000 7f0e51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:32.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.142+0000 7f0e51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:32.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.142+0000 7f0e51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:32.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.142+0000 7f0e51ffb640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:43:32.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.142+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c06c8e0 con 0x7f0e5c0b12d0 2024-03-25T19:43:32.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.150+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 450256717 0 0) 0x7f0e40004eb0 con 0x7f0e5c0aedf0 2024-03-25T19:43:32.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.150+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e5c1c4750 con 0x7f0e5c0aedf0 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.150+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0e4c001550 con 0x7f0e5c0aedf0 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.150+0000 7f0e51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.150+0000 7f0e51ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.150+0000 7f0e51ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:32.151 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:32.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.155+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 361792844 0 0) 0x7f0e5c1c3680 con 0x7f0e5c06c380 2024-03-25T19:43:32.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 >> v1:172.21.15.169:6789/0 conn(0x7f0e5c0b12d0 legacy=0x7f0e5c1d79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 >> v1:172.21.15.69:6789/0 conn(0x7f0e5c0aedf0 legacy=0x7f0e5c1c0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:32.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 1 monclient: found mon.c 2024-03-25T19:43:32.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:32.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c1c2820 con 0x7f0e5c06c380 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 5 monclient: authenticate success, global_id 4334 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 10 monclient: _renew_subs 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c1c1710 con 0x7f0e5c06c380 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 10 monclient: _renew_subs 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c1c1300 con 0x7f0e5c06c380 2024-03-25T19:43:32.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.156+0000 7f0e6390e640 1 librados: init done 2024-03-25T19:43:32.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.157+0000 7f0e6390e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:32.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.157+0000 7f0e6390e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:32.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.157+0000 7f0e6390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:32.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.157+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c1c4990 con 0x7f0e5c06c380 2024-03-25T19:43:32.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.444+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e58003610 con 0x7f0e5c06c380 2024-03-25T19:43:32.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.444+0000 7f0e51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:32.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.444+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0e58003880 con 0x7f0e5c06c380 2024-03-25T19:43:32.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.444+0000 7f0e51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:32.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.444+0000 7f0e51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:32.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.444+0000 7f0e51ffb640 10 monclient: dump: 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.444+0000 7f0e537fe640 4 set_mon_vals no callback set 2024-03-25T19:43:32.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.445+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f0e58003aa0 con 0x7f0e5c06c380 2024-03-25T19:43:32.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.446+0000 7f0e60ea9640 10 monclient: get_auth_request con 0x7f0e4006ddd0 auth_method 0 2024-03-25T19:43:32.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.446+0000 7f0e51ffb640 10 client.4334.objecter ms_handle_connect 0x7f0e4006ddd0 2024-03-25T19:43:32.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.451+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f0e580533a0 con 0x7f0e5c06c380 2024-03-25T19:43:32.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.451+0000 7f0e51ffb640 10 client.4334.objecter ms_dispatch 0x7f0e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:32.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.451+0000 7f0e51ffb640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:32.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.451+0000 7f0e51ffb640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:32.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.451+0000 7f0e51ffb640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-25T19:43:32.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.451+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e5804eeb0 con 0x7f0e5c06c380 2024-03-25T19:43:32.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.451+0000 7f0e51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:32.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.451+0000 7f0e51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:32.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.664+0000 7f0e6390e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:32.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.664+0000 7f0e6390e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:32.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.664+0000 7f0e6390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:32.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.664+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e5c1c34c0 con 0x7f0e5c06c380 2024-03-25T19:43:32.700 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.700+0000 7f0e51ffb640 1 -- 172.21.15.69:0/3094213285 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f0e58050de0 con 0x7f0e5c06c380 2024-03-25T19:43:32.700 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.700+0000 7f0e51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:32.700 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.700+0000 7f0e51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:32.700 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:43:32.700 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:32.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 10 librados: watch_flush enter 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 10 librados: watch_flush exit 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 10 client.4334.objecter successfully canceled tick 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 >> v1:172.21.15.169:6800/3072548259 conn(0x7f0e4006ddd0 legacy=0x7f0e40070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 10 monclient: shutdown 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 >> v1:172.21.15.69:6790/0 conn(0x7f0e5c06c380 legacy=0x7f0e5c1c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 shutdown_connections 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 >> 172.21.15.69:0/3094213285 conn(0x7f0e5c0b5d10 msgr2=0x7f0e5c0ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 shutdown_connections 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 1 -- 172.21.15.69:0/3094213285 wait complete. 2024-03-25T19:43:32.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:32.702+0000 7f0e6390e640 1 librados: shutdown 2024-03-25T19:43:32.714 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:39.521 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:39.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.626+0000 7f22854ba640 10 monclient: get_monmap_and_config 2024-03-25T19:43:39.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.626+0000 7f22854ba640 10 monclient: build_initial_monmap 2024-03-25T19:43:39.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.627+0000 7f22854ba640 10 monclient: monmap: 2024-03-25T19:43:39.627 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:39.627 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:39.627 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:43:39.628222+0000 2024-03-25T19:43:39.628 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:43:39.628222+0000 2024-03-25T19:43:39.628 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:39.628 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:39.628 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:39.628 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:43:39.628 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:43:39.628 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.627+0000 7f22854ba640 1 Processor -- start 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.628+0000 7f22854ba640 1 -- start start 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.628+0000 7f22854ba640 10 monclient: init 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.628+0000 7f22854ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.628+0000 7f22854ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.628+0000 7f22854ba640 10 monclient(hunting): picked mon.noname-b con 0x7f22800a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.629+0000 7f22854ba640 10 monclient(hunting): picked mon.noname-a con 0x7f228013f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.629+0000 7f22854ba640 10 monclient(hunting): picked mon.noname-c con 0x7f22801430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.629+0000 7f22854ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2280142520 con 0x7f228013f710 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.629+0000 7f22854ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800b28a0 con 0x7f22801430c0 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.629+0000 7f22854ba640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800b2a30 con 0x7f22800a98f0 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.629+0000 7f22854ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.629+0000 7f22854ba640 10 monclient(hunting): authenticate will time out at 5031.897752s 2024-03-25T19:43:39.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.637+0000 7f2277fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f228013f710 0x7f22801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47884/0 (socket says 172.21.15.69:47884) 2024-03-25T19:43:39.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.637+0000 7f2277fff640 1 -- 172.21.15.69:0/1602578461 learned_addr learned my addr 172.21.15.69:0/1602578461 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:39.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.645+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671203216 0 0) 0x7f2280142520 con 0x7f228013f710 2024-03-25T19:43:39.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.645+0000 7f227e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.645+0000 7f227e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.645+0000 7f227e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.645+0000 7f227e7fc640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:43:39.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.645+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226c003520 con 0x7f228013f710 2024-03-25T19:43:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.650+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272833219 0 0) 0x7f22800b2a30 con 0x7f22800a98f0 2024-03-25T19:43:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.650+0000 7f227e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.650+0000 7f227e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.650+0000 7f227e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.650+0000 7f227e7fc640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:43:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.650+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2280142520 con 0x7f22800a98f0 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.654+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067294241 0 0) 0x7f22800b28a0 con 0x7f22801430c0 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.654+0000 7f227e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.654+0000 7f227e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.654+0000 7f227e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.654+0000 7f227e7fc640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.654+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22800b2a30 con 0x7f22801430c0 2024-03-25T19:43:39.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977008962 0 0) 0x7f2280142520 con 0x7f22800a98f0 2024-03-25T19:43:39.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 >> v1:172.21.15.169:6789/0 conn(0x7f22801430c0 legacy=0x7f228014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 >> v1:172.21.15.69:6789/0 conn(0x7f228013f710 legacy=0x7f22801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22800b44c0 con 0x7f22800a98f0 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f22854ba640 5 monclient: authenticate success, global_id 4337 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f22700019f0 con 0x7f22800a98f0 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:39.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f227e7fc640 10 monclient: dump: 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:39.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.655+0000 7f22854ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.663+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2270001b50 con 0x7f22800a98f0 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.663+0000 7f227e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.663+0000 7f227e7fc640 1 -- 172.21.15.69:0/1602578461 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f2270001f60 con 0x7f22800a98f0 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.663+0000 7f22854ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.663+0000 7f22854ba640 4 set_mon_vals no callback set 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.663+0000 7f227e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.663+0000 7f227e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.663+0000 7f227e7fc640 10 monclient: dump: 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:39.663 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:39.664 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:39.664 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:39.664 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.664 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:39.664 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:39.664 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:39.664 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:39.664 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:39.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 10 monclient: shutdown 2024-03-25T19:43:39.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:39.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 1 -- 172.21.15.69:0/1602578461 >> v1:172.21.15.69:6790/0 conn(0x7f22800a98f0 legacy=0x7f22800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 1 -- 172.21.15.69:0/1602578461 shutdown_connections 2024-03-25T19:43:39.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 1 -- 172.21.15.69:0/1602578461 >> 172.21.15.69:0/1602578461 conn(0x7f22800b7ea0 msgr2=0x7f22800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 1 -- 172.21.15.69:0/1602578461 shutdown_connections 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 1 -- 172.21.15.69:0/1602578461 wait complete. 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 10 monclient: build_initial_monmap 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.664+0000 7f22854ba640 10 monclient: monmap: 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.665+0000 7f22854ba640 1 librados: starting msgr at 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.665+0000 7f22854ba640 1 librados: starting objecter 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.665+0000 7f22854ba640 1 Processor -- start 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.665+0000 7f22854ba640 1 -- start start 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.665+0000 7f22854ba640 1 librados: setting wanted keys 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.665+0000 7f22854ba640 1 librados: calling monclient init 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.665+0000 7f22854ba640 10 monclient: init 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 10 monclient(hunting): picked mon.noname-a con 0x7f22800a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 10 monclient(hunting): picked mon.noname-c con 0x7f228013f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 10 monclient(hunting): picked mon.noname-b con 0x7f22801430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800abc80 con 0x7f22800a98f0 2024-03-25T19:43:39.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800abf60 con 0x7f22801430c0 2024-03-25T19:43:39.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800ac240 con 0x7f228013f710 2024-03-25T19:43:39.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:39.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.666+0000 7f22854ba640 10 monclient(hunting): authenticate will time out at 5031.934750s 2024-03-25T19:43:39.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.671+0000 7f227f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f22801430c0 0x7f22801e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:53530/0 (socket says 172.21.15.69:53530) 2024-03-25T19:43:39.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.671+0000 7f227f7fe640 1 -- 172.21.15.69:0/1264221225 learned_addr learned my addr 172.21.15.69:0/1264221225 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:39.672 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.672+0000 7f22777fe640 10 client.?.objecter ms_handle_connect 0x7f22801430c0 2024-03-25T19:43:39.672 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.672+0000 7f22777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:39.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.674+0000 7f22777fe640 10 client.?.objecter ms_handle_connect 0x7f22800a98f0 2024-03-25T19:43:39.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.674+0000 7f22777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:39.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.676+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665327400 0 0) 0x7f22800abf60 con 0x7f22801430c0 2024-03-25T19:43:39.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.676+0000 7f22777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.676+0000 7f22777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.676+0000 7f22777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.676+0000 7f22777fe640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:43:39.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.676+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2260004eb0 con 0x7f22801430c0 2024-03-25T19:43:39.682 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.682+0000 7f22777fe640 10 client.?.objecter ms_handle_connect 0x7f228013f710 2024-03-25T19:43:39.682 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.682+0000 7f22777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:39.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.685+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228283258 0 0) 0x7f22800abc80 con 0x7f22800a98f0 2024-03-25T19:43:39.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.685+0000 7f22777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.685+0000 7f22777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.685+0000 7f22777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.685+0000 7f22777fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:43:39.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.685+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22800abf60 con 0x7f22800a98f0 2024-03-25T19:43:39.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.689+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368952116 0 0) 0x7f2260004eb0 con 0x7f22801430c0 2024-03-25T19:43:39.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.689+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22800abc80 con 0x7f22801430c0 2024-03-25T19:43:39.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.689+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f22780018f0 con 0x7f22801430c0 2024-03-25T19:43:39.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.689+0000 7f22777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.689+0000 7f22777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.689+0000 7f22777fe640 10 monclient(hunting): dump: 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:39.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.689+0000 7f22777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:39.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1770287689 0 0) 0x7f22800abc80 con 0x7f22801430c0 2024-03-25T19:43:39.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 >> v1:172.21.15.69:6790/0 conn(0x7f228013f710 legacy=0x7f22801365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 >> v1:172.21.15.69:6789/0 conn(0x7f22800a98f0 legacy=0x7f22800ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:39.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 1 monclient: found mon.b 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2280138520 con 0x7f22801430c0 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.693+0000 7f22777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 10 monclient: _renew_subs 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2280137f50 con 0x7f22801430c0 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:39.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 10 monclient: _renew_subs 2024-03-25T19:43:39.696 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:39.696 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2280138e20 con 0x7f22801430c0 2024-03-25T19:43:39.696 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.694+0000 7f22854ba640 1 librados: init done 2024-03-25T19:43:39.696 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.695+0000 7f22854ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:39.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.695+0000 7f22854ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:39.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.695+0000 7f22854ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:39.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.695+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22800b3ef0 con 0x7f22801430c0 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.714+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22780035f0 con 0x7f22801430c0 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.714+0000 7f22777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.714+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f2278003a00 con 0x7f22801430c0 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.714+0000 7f22777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.714+0000 7f22777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.714+0000 7f22777fe640 10 monclient: dump: 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:39.714 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:39.715 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.715 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:39.715 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:39.715 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:39.715 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:39.715 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:39.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.714+0000 7f227dffb640 4 set_mon_vals no callback set 2024-03-25T19:43:39.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.715+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f2278003c20 con 0x7f22801430c0 2024-03-25T19:43:39.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.715+0000 7f227effd640 10 monclient: get_auth_request con 0x7f226006dd10 auth_method 0 2024-03-25T19:43:39.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.716+0000 7f22777fe640 10 client.4336.objecter ms_handle_connect 0x7f226006dd10 2024-03-25T19:43:39.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.717+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f2278054740 con 0x7f22801430c0 2024-03-25T19:43:39.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.717+0000 7f22777fe640 10 client.4336.objecter ms_dispatch 0x7f2280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:39.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.717+0000 7f22777fe640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:39.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.717+0000 7f22777fe640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:39.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.717+0000 7f22777fe640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T19:43:39.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.983+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2278050250 con 0x7f22801430c0 2024-03-25T19:43:39.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.983+0000 7f22777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:39.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:39.983+0000 7f22777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:40.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.235+0000 7f22854ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:40.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.235+0000 7f22854ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:40.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.235+0000 7f22854ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:40.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.236+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22800627b0 con 0x7f22801430c0 2024-03-25T19:43:40.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.265+0000 7f22777fe640 1 -- 172.21.15.69:0/1264221225 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f2278052180 con 0x7f22801430c0 2024-03-25T19:43:40.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.265+0000 7f22777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:40.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.265+0000 7f22777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:40.266 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:43:40.266 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:40.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 10 librados: watch_flush enter 2024-03-25T19:43:40.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 10 librados: watch_flush exit 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 10 client.4336.objecter successfully canceled tick 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 >> v1:172.21.15.169:6800/3072548259 conn(0x7f226006dd10 legacy=0x7f22600701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 10 monclient: shutdown 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 >> v1:172.21.15.169:6789/0 conn(0x7f22801430c0 legacy=0x7f22801e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 shutdown_connections 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 >> 172.21.15.69:0/1264221225 conn(0x7f22800b7ea0 msgr2=0x7f2280074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 shutdown_connections 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 1 -- 172.21.15.69:0/1264221225 wait complete. 2024-03-25T19:43:40.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:40.268+0000 7f22854ba640 1 librados: shutdown 2024-03-25T19:43:40.280 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:41.717 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:19.681 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:41.733 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:19.697 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:41.768 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:19.732 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:41.779 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:19.733 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:41.781 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:19.745 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:41.793 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:19.747 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:41.799 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:19.763 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:41.816 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:19.770 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:41.823 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:19.787 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.087 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.203+0000 7f89c10d9640 10 monclient: get_monmap_and_config 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.203+0000 7f89c10d9640 10 monclient: build_initial_monmap 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.203+0000 7f89c10d9640 10 monclient: monmap: 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:43:47.204409+0000 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:43:47.204409+0000 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:47.203 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:47.204 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:47.204 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:43:47.204 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:43:47.204 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:47.204 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.203+0000 7f89c10d9640 1 Processor -- start 2024-03-25T19:43:47.204 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.203+0000 7f89c10d9640 1 -- start start 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.203+0000 7f89c10d9640 10 monclient: init 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 10 monclient(hunting): picked mon.noname-a con 0x7f89bc0a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 10 monclient(hunting): picked mon.noname-c con 0x7f89bc13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 10 monclient(hunting): picked mon.noname-b con 0x7f89bc1430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc142520 con 0x7f89bc0a98f0 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc0b28a0 con 0x7f89bc13f710 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc0b2a30 con 0x7f89bc1430c0 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:47.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.204+0000 7f89c10d9640 10 monclient(hunting): authenticate will time out at 5039.473547s 2024-03-25T19:43:47.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.220+0000 7f89bb59e640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f89bc1430c0 0x7f89bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:52584/0 (socket says 172.21.15.69:52584) 2024-03-25T19:43:47.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.220+0000 7f89bb59e640 1 -- 172.21.15.69:0/2070250645 learned_addr learned my addr 172.21.15.69:0/2070250645 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:47.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.237+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025670001 0 0) 0x7f89bc0b2a30 con 0x7f89bc1430c0 2024-03-25T19:43:47.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.237+0000 7f89b9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.237+0000 7f89b9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.237+0000 7f89b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.237+0000 7f89b9d9b640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:43:47.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.237+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a4003520 con 0x7f89bc1430c0 2024-03-25T19:43:47.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.241+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105284560 0 0) 0x7f89bc0b28a0 con 0x7f89bc13f710 2024-03-25T19:43:47.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.241+0000 7f89b9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.241+0000 7f89b9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.241+0000 7f89b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.241+0000 7f89b9d9b640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:43:47.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.241+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89bc0b2a30 con 0x7f89bc13f710 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 801227663 0 0) 0x7f89a4003520 con 0x7f89bc1430c0 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 >> v1:172.21.15.169:6789/0 conn(0x7f89bc13f710 legacy=0x7f89bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 >> v1:172.21.15.69:6789/0 conn(0x7f89bc0a98f0 legacy=0x7f89bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 1 monclient: found mon.noname-b 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0b44c0 con 0x7f89bc1430c0 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f89b4001a10 con 0x7f89bc1430c0 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89c10d9640 5 monclient: authenticate success, global_id 4343 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89c10d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89b9d9b640 10 monclient: dump: 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.250+0000 7f89c10d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:47.259 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.259+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89b4001b70 con 0x7f89bc1430c0 2024-03-25T19:43:47.259 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.259+0000 7f89b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:47.259 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.259+0000 7f89b9d9b640 1 -- 172.21.15.69:0/2070250645 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f89b4001f80 con 0x7f89bc1430c0 2024-03-25T19:43:47.259 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.259+0000 7f89b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.259 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.259+0000 7f89b9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.259+0000 7f89b9d9b640 10 monclient: dump: 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.259+0000 7f89c10d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:47.260 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.259+0000 7f89c10d9640 4 set_mon_vals no callback set 2024-03-25T19:43:47.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.260+0000 7f89c10d9640 10 monclient: shutdown 2024-03-25T19:43:47.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.260+0000 7f89c10d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.260+0000 7f89c10d9640 1 -- 172.21.15.69:0/2070250645 >> v1:172.21.15.69:6790/0 conn(0x7f89bc1430c0 legacy=0x7f89bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.260+0000 7f89c10d9640 1 -- 172.21.15.69:0/2070250645 shutdown_connections 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.260+0000 7f89c10d9640 1 -- 172.21.15.69:0/2070250645 >> 172.21.15.69:0/2070250645 conn(0x7f89bc0b7ea0 msgr2=0x7f89bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.260+0000 7f89c10d9640 1 -- 172.21.15.69:0/2070250645 shutdown_connections 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.260+0000 7f89c10d9640 1 -- 172.21.15.69:0/2070250645 wait complete. 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.261+0000 7f89c10d9640 10 monclient: build_initial_monmap 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.261+0000 7f89c10d9640 10 monclient: monmap: 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:47.263 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:43:47.263 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:47.263 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:43:47.263 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:43:47.263 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.262+0000 7f89c10d9640 1 librados: starting msgr at 2024-03-25T19:43:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.262+0000 7f89c10d9640 1 librados: starting objecter 2024-03-25T19:43:47.264 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.262+0000 7f89c10d9640 1 Processor -- start 2024-03-25T19:43:47.264 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.262+0000 7f89c10d9640 1 -- start start 2024-03-25T19:43:47.264 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.262+0000 7f89c10d9640 1 librados: setting wanted keys 2024-03-25T19:43:47.264 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.262+0000 7f89c10d9640 1 librados: calling monclient init 2024-03-25T19:43:47.264 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.262+0000 7f89c10d9640 10 monclient: init 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 10 monclient(hunting): picked mon.noname-a con 0x7f89bc0a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 10 monclient(hunting): picked mon.noname-b con 0x7f89bc13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 10 monclient(hunting): picked mon.noname-c con 0x7f89bc1430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc136ad0 con 0x7f89bc0a98f0 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc0b7520 con 0x7f89bc13f710 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc135810 con 0x7f89bc1430c0 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.263+0000 7f89c10d9640 10 monclient(hunting): authenticate will time out at 5039.531802s 2024-03-25T19:43:47.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.267+0000 7f89bb59e640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f89bc1430c0 0x7f89bc130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:52628/0 (socket says 172.21.15.69:52628) 2024-03-25T19:43:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.268+0000 7f89bb59e640 1 -- 172.21.15.69:0/131481508 learned_addr learned my addr 172.21.15.69:0/131481508 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.268+0000 7f89a37fe640 10 client.?.objecter ms_handle_connect 0x7f89bc1430c0 2024-03-25T19:43:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.268+0000 7f89a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:47.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.272+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635033462 0 0) 0x7f89bc135810 con 0x7f89bc1430c0 2024-03-25T19:43:47.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.272+0000 7f89a37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.272+0000 7f89a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.272+0000 7f89a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.272+0000 7f89a37fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:43:47.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.273+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a8004eb0 con 0x7f89bc1430c0 2024-03-25T19:43:47.275 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.275+0000 7f89a37fe640 10 client.?.objecter ms_handle_connect 0x7f89bc0a98f0 2024-03-25T19:43:47.275 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.275+0000 7f89a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:47.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.284+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 751933569 0 0) 0x7f89a8004eb0 con 0x7f89bc1430c0 2024-03-25T19:43:47.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.284+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89bc135810 con 0x7f89bc1430c0 2024-03-25T19:43:47.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.284+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f89b4001ec0 con 0x7f89bc1430c0 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.284+0000 7f89a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.284+0000 7f89a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.284+0000 7f89a37fe640 10 monclient(hunting): dump: 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:47.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.284+0000 7f89a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:47.286 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.285+0000 7f89a37fe640 10 client.?.objecter ms_handle_connect 0x7f89bc13f710 2024-03-25T19:43:47.286 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.285+0000 7f89a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:47.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106762908 0 0) 0x7f89bc136ad0 con 0x7f89bc0a98f0 2024-03-25T19:43:47.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:43:47.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a8004eb0 con 0x7f89bc0a98f0 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 506545352 0 0) 0x7f89bc135810 con 0x7f89bc1430c0 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 >> v1:172.21.15.169:6789/0 conn(0x7f89bc13f710 legacy=0x7f89bc0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 >> v1:172.21.15.69:6789/0 conn(0x7f89bc0a98f0 legacy=0x7f89bc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 1 monclient: found mon.c 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0b6180 con 0x7f89bc1430c0 2024-03-25T19:43:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89c10d9640 5 monclient: authenticate success, global_id 4346 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89c10d9640 10 monclient: _renew_subs 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89c10d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.289+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc0b7a40 con 0x7f89bc1430c0 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 10 monclient: _renew_subs 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc131840 con 0x7f89bc1430c0 2024-03-25T19:43:47.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 1 librados: init done 2024-03-25T19:43:47.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:47.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:47.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:47.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.290+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc136f60 con 0x7f89bc1430c0 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.583+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89b4004e10 con 0x7f89bc1430c0 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.583+0000 7f89a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.583+0000 7f89b959a640 4 set_mon_vals no callback set 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.584+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f89b4002020 con 0x7f89bc1430c0 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.584+0000 7f89a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.584+0000 7f89a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.584+0000 7f89a37fe640 10 monclient: dump: 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:47.584 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:47.585 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.585 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:47.585 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:47.585 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:47.585 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:47.585 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:47.585 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.584+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f89b401adf0 con 0x7f89bc1430c0 2024-03-25T19:43:47.586 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.585+0000 7f89bad9d640 10 monclient: get_auth_request con 0x7f89a806dd90 auth_method 0 2024-03-25T19:43:47.586 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.585+0000 7f89a37fe640 10 client.4346.objecter ms_handle_connect 0x7f89a806dd90 2024-03-25T19:43:47.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.591+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f89b40549e0 con 0x7f89bc1430c0 2024-03-25T19:43:47.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.591+0000 7f89a37fe640 10 client.4346.objecter ms_dispatch 0x7f89bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:47.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.591+0000 7f89a37fe640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:47.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.591+0000 7f89a37fe640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:47.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.591+0000 7f89a37fe640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-25T19:43:47.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.591+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89b4050320 con 0x7f89bc1430c0 2024-03-25T19:43:47.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.591+0000 7f89a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:47.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.591+0000 7f89a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:47.860 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.860+0000 7f89c10d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:47.860 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.860+0000 7f89c10d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:47.860 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.860+0000 7f89c10d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:43:47.860 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.860+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89bc0b6310 con 0x7f89bc1430c0 2024-03-25T19:43:47.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.884+0000 7f89a37fe640 1 -- 172.21.15.69:0/131481508 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f89b4052420 con 0x7f89bc1430c0 2024-03-25T19:43:47.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.884+0000 7f89a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:47.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.884+0000 7f89a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:47.884 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:43:47.885 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 10 librados: watch_flush enter 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 10 librados: watch_flush exit 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 10 client.4346.objecter successfully canceled tick 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 >> v1:172.21.15.169:6800/3072548259 conn(0x7f89a806dd90 legacy=0x7f89a8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 10 monclient: shutdown 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 >> v1:172.21.15.69:6790/0 conn(0x7f89bc1430c0 legacy=0x7f89bc130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 shutdown_connections 2024-03-25T19:43:47.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.886+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 >> 172.21.15.69:0/131481508 conn(0x7f89bc0b7ea0 msgr2=0x7f89bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:47.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.887+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 shutdown_connections 2024-03-25T19:43:47.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.887+0000 7f89c10d9640 1 -- 172.21.15.69:0/131481508 wait complete. 2024-03-25T19:43:47.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:47.887+0000 7f89c10d9640 1 librados: shutdown 2024-03-25T19:43:47.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:54.705 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.823+0000 7f67d9baf640 10 monclient: get_monmap_and_config 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.823+0000 7f67d9baf640 10 monclient: build_initial_monmap 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.823+0000 7f67d9baf640 10 monclient: monmap: 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:43:54.824381+0000 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:43:54.824381+0000 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:43:54.823 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:43:54.824 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:54.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.823+0000 7f67d9baf640 1 Processor -- start 2024-03-25T19:43:54.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 1 -- start start 2024-03-25T19:43:54.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 10 monclient: init 2024-03-25T19:43:54.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:54.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:54.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 10 monclient(hunting): picked mon.noname-a con 0x7f67d40a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:54.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 10 monclient(hunting): picked mon.noname-c con 0x7f67d413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:54.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 10 monclient(hunting): picked mon.noname-b con 0x7f67d41430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:54.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d4142520 con 0x7f67d40a98f0 2024-03-25T19:43:54.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40b28a0 con 0x7f67d413f710 2024-03-25T19:43:54.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40b2a30 con 0x7f67d41430c0 2024-03-25T19:43:54.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:54.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.824+0000 7f67d9baf640 10 monclient(hunting): authenticate will time out at 5047.093509s 2024-03-25T19:43:54.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.833+0000 7f67d3fff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f67d41430c0 0x7f67d414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:59258/0 (socket says 172.21.15.69:59258) 2024-03-25T19:43:54.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.833+0000 7f67d3fff640 1 -- 172.21.15.69:0/606431163 learned_addr learned my addr 172.21.15.69:0/606431163 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:54.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.848+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290444179 0 0) 0x7f67d40b2a30 con 0x7f67d41430c0 2024-03-25T19:43:54.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.848+0000 7f67d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.848+0000 7f67d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.848+0000 7f67d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.848+0000 7f67d27fc640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:43:54.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.848+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67bc003520 con 0x7f67d41430c0 2024-03-25T19:43:54.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.850+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271954123 0 0) 0x7f67d4142520 con 0x7f67d40a98f0 2024-03-25T19:43:54.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.850+0000 7f67d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.850+0000 7f67d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.850+0000 7f67d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.850+0000 7f67d27fc640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:43:54.850 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.850+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d40b2a30 con 0x7f67d40a98f0 2024-03-25T19:43:54.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.853+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628715608 0 0) 0x7f67d40b28a0 con 0x7f67d413f710 2024-03-25T19:43:54.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.853+0000 7f67d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.853+0000 7f67d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.853+0000 7f67d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.853+0000 7f67d27fc640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:43:54.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.853+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d4142520 con 0x7f67d413f710 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740787374 0 0) 0x7f67d40b2a30 con 0x7f67d40a98f0 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 >> v1:172.21.15.69:6790/0 conn(0x7f67d41430c0 legacy=0x7f67d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 >> v1:172.21.15.169:6789/0 conn(0x7f67d413f710 legacy=0x7f67d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 1 monclient: found mon.noname-a 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d40b44c0 con 0x7f67d40a98f0 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f67c4001670 con 0x7f67d40a98f0 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d27fc640 10 monclient: dump: 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:54.860 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:54.860 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:54.860 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d9baf640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:43:54.860 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.857+0000 7f67d9baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.880+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67c40017d0 con 0x7f67d40a98f0 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.880+0000 7f67d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.880+0000 7f67d27fc640 1 -- 172.21.15.69:0/606431163 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f67c4001be0 con 0x7f67d40a98f0 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.880+0000 7f67d9baf640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.880+0000 7f67d9baf640 4 set_mon_vals no callback set 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.880+0000 7f67d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.880+0000 7f67d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.880+0000 7f67d27fc640 10 monclient: dump: 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:54.880 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:54.881 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.881 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.881 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.881 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:54.881 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:54.881 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:54.881 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:54.881 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 10 monclient: shutdown 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 1 -- 172.21.15.69:0/606431163 >> v1:172.21.15.69:6789/0 conn(0x7f67d40a98f0 legacy=0x7f67d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 1 -- 172.21.15.69:0/606431163 shutdown_connections 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 1 -- 172.21.15.69:0/606431163 >> 172.21.15.69:0/606431163 conn(0x7f67d40b7ea0 msgr2=0x7f67d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 1 -- 172.21.15.69:0/606431163 shutdown_connections 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 1 -- 172.21.15.69:0/606431163 wait complete. 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 10 monclient: build_initial_monmap 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.881+0000 7f67d9baf640 10 monclient: monmap: 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:43:54.883 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.882+0000 7f67d9baf640 1 librados: starting msgr at 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.882+0000 7f67d9baf640 1 librados: starting objecter 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.882+0000 7f67d9baf640 1 Processor -- start 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.882+0000 7f67d9baf640 1 -- start start 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.882+0000 7f67d9baf640 1 librados: setting wanted keys 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.882+0000 7f67d9baf640 1 librados: calling monclient init 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.882+0000 7f67d9baf640 10 monclient: init 2024-03-25T19:43:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 10 monclient(hunting): picked mon.noname-b con 0x7f67d40a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 10 monclient(hunting): picked mon.noname-c con 0x7f67d413f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 10 monclient(hunting): picked mon.noname-a con 0x7f67d41430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d4138f10 con 0x7f67d41430c0 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d4139970 con 0x7f67d40a98f0 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40abb50 con 0x7f67d413f710 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.883+0000 7f67d9baf640 10 monclient(hunting): authenticate will time out at 5047.151805s 2024-03-25T19:43:54.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.891+0000 7f67d2ffd640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f67d413f710 0x7f67d40ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:59300/0 (socket says 172.21.15.69:59300) 2024-03-25T19:43:54.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.891+0000 7f67d2ffd640 1 -- 172.21.15.69:0/1614200090 learned_addr learned my addr 172.21.15.69:0/1614200090 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:43:54.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.898+0000 7f67bbfff640 10 client.?.objecter ms_handle_connect 0x7f67d40a98f0 2024-03-25T19:43:54.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.898+0000 7f67bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.901+0000 7f67bbfff640 10 client.?.objecter ms_handle_connect 0x7f67d41430c0 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.901+0000 7f67bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:54.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.908+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244270492 0 0) 0x7f67d4139970 con 0x7f67d40a98f0 2024-03-25T19:43:54.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.908+0000 7f67bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.908+0000 7f67bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.908+0000 7f67bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.908+0000 7f67bbfff640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:43:54.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.908+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c0004eb0 con 0x7f67d40a98f0 2024-03-25T19:43:54.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.911+0000 7f67bbfff640 10 client.?.objecter ms_handle_connect 0x7f67d413f710 2024-03-25T19:43:54.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.911+0000 7f67bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:54.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.914+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070018144 0 0) 0x7f67d4138f10 con 0x7f67d41430c0 2024-03-25T19:43:54.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.914+0000 7f67bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.914+0000 7f67bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.914+0000 7f67bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.914+0000 7f67bbfff640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:43:54.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.914+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d4139970 con 0x7f67d41430c0 2024-03-25T19:43:54.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.918+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3971179701 0 0) 0x7f67c0004eb0 con 0x7f67d40a98f0 2024-03-25T19:43:54.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.918+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d4138f10 con 0x7f67d40a98f0 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.923+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155190958 0 0) 0x7f67d40abb50 con 0x7f67d413f710 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.923+0000 7f67bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.923+0000 7f67bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.923+0000 7f67bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.923+0000 7f67bbfff640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:43:54.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.923+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c0004eb0 con 0x7f67d413f710 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.925+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f67c4001be0 con 0x7f67d40a98f0 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.925+0000 7f67bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.925+0000 7f67bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.925+0000 7f67bbfff640 10 monclient(hunting): dump: 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.925+0000 7f67bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:54.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.931+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1046945051 0 0) 0x7f67d4139970 con 0x7f67d41430c0 2024-03-25T19:43:54.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.931+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d40abb50 con 0x7f67d41430c0 2024-03-25T19:43:54.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.931+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f67cc001a70 con 0x7f67d41430c0 2024-03-25T19:43:54.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.931+0000 7f67bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.931+0000 7f67bbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:54.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.931+0000 7f67bbfff640 10 monclient(hunting): dump: 2024-03-25T19:43:54.931 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:54.931 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:54.932 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.932 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.932 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.932 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:54.932 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:54.932 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:54.932 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:54.932 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:54.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1062479448 0 0) 0x7f67d4138f10 con 0x7f67d40a98f0 2024-03-25T19:43:54.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 >> v1:172.21.15.69:6790/0 conn(0x7f67d413f710 legacy=0x7f67d40ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 >> v1:172.21.15.69:6789/0 conn(0x7f67d41430c0 legacy=0x7f67d4138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:54.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 1 monclient: found mon.b 2024-03-25T19:43:54.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:54.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d40abe30 con 0x7f67d40a98f0 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 10 monclient: _finish_auth 0 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.932+0000 7f67d9baf640 5 monclient: authenticate success, global_id 4348 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.933+0000 7f67d9baf640 10 monclient: _renew_subs 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.933+0000 7f67d9baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.933+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d4137b40 con 0x7f67d40a98f0 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.933+0000 7f67d9baf640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.933+0000 7f67d9baf640 10 monclient: _renew_subs 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.933+0000 7f67d9baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.933+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d4138c10 con 0x7f67d40a98f0 2024-03-25T19:43:54.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.933+0000 7f67d9baf640 1 librados: init done 2024-03-25T19:43:54.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.934+0000 7f67d9baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:54.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.934+0000 7f67d9baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:54.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.934+0000 7f67d9baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:54.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.934+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d40627b0 con 0x7f67d40a98f0 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.951+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67c40038a0 con 0x7f67d40a98f0 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.951+0000 7f67bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.951+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f67c4003bb0 con 0x7f67d40a98f0 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.951+0000 7f67bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.951+0000 7f67bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.951+0000 7f67bbfff640 10 monclient: dump: 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:43:54.951 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.952 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:43:54.952 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:43:54.952 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:43:54.952 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:43:54.952 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:43:54.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.951+0000 7f67d1ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:54.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.952+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f67c4019b80 con 0x7f67d40a98f0 2024-03-25T19:43:54.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.952+0000 7f67d2ffd640 10 monclient: get_auth_request con 0x7f67c006df90 auth_method 0 2024-03-25T19:43:54.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.953+0000 7f67bbfff640 10 client.4348.objecter ms_handle_connect 0x7f67c006df90 2024-03-25T19:43:54.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.954+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f67c4053790 con 0x7f67d40a98f0 2024-03-25T19:43:54.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.954+0000 7f67bbfff640 10 client.4348.objecter ms_dispatch 0x7f67d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:54.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.954+0000 7f67bbfff640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:54.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.954+0000 7f67bbfff640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:54.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:54.954+0000 7f67bbfff640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T19:43:55.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.221+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67c404f0d0 con 0x7f67d40a98f0 2024-03-25T19:43:55.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.221+0000 7f67bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:55.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.221+0000 7f67bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:55.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.466+0000 7f67d9baf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:55.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.466+0000 7f67d9baf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:55.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.466+0000 7f67d9baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:43:55.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.466+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67d413db40 con 0x7f67d40a98f0 2024-03-25T19:43:55.484 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.484+0000 7f67bbfff640 1 -- 172.21.15.69:0/1614200090 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f67c40511d0 con 0x7f67d40a98f0 2024-03-25T19:43:55.484 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.484+0000 7f67bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:55.484 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.484+0000 7f67bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:55.485 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:43:55.485 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:55.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 10 librados: watch_flush enter 2024-03-25T19:43:55.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 10 librados: watch_flush exit 2024-03-25T19:43:55.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T19:43:55.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 10 client.4348.objecter successfully canceled tick 2024-03-25T19:43:55.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 >> v1:172.21.15.169:6800/3072548259 conn(0x7f67c006df90 legacy=0x7f67c0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:55.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 10 monclient: shutdown 2024-03-25T19:43:55.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:55.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 >> v1:172.21.15.169:6789/0 conn(0x7f67d40a98f0 legacy=0x7f67d4136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:55.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 shutdown_connections 2024-03-25T19:43:55.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 >> 172.21.15.69:0/1614200090 conn(0x7f67d40b7ea0 msgr2=0x7f67d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:55.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 shutdown_connections 2024-03-25T19:43:55.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 1 -- 172.21.15.69:0/1614200090 wait complete. 2024-03-25T19:43:55.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:43:55.487+0000 7f67d9baf640 1 librados: shutdown 2024-03-25T19:43:55.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:01.786 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:39.750 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.836 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:39.800 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.867 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:39.831 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.912 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:39.876 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.913 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:39.867 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.918 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:39.873 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.919 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:39.883 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.949 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:39.903 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.951 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:39.915 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:02.308 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:02.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.415+0000 7f4a97c74640 10 monclient: get_monmap_and_config 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.415+0000 7f4a97c74640 10 monclient: build_initial_monmap 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.415+0000 7f4a97c74640 10 monclient: monmap: 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:44:02.417096+0000 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:44:02.417096+0000 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:44:02.416 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.416+0000 7f4a97c74640 1 Processor -- start 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.416+0000 7f4a97c74640 1 -- start start 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.416+0000 7f4a97c74640 10 monclient: init 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 10 monclient(hunting): picked mon.noname-a con 0x7f4a900b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 10 monclient(hunting): picked mon.noname-c con 0x7f4a900b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 10 monclient(hunting): picked mon.noname-b con 0x7f4a9014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:02.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a900af940 con 0x7f4a900b4d60 2024-03-25T19:44:02.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a900b0a70 con 0x7f4a900b1730 2024-03-25T19:44:02.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a900b0d50 con 0x7f4a9014be90 2024-03-25T19:44:02.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:02.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.417+0000 7f4a97c74640 10 monclient(hunting): authenticate will time out at 5054.686305s 2024-03-25T19:44:02.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.423+0000 7f4a95a10640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4a900b4d60 0x7f4a9013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47892/0 (socket says 172.21.15.69:47892) 2024-03-25T19:44:02.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.423+0000 7f4a95a10640 1 -- 172.21.15.69:0/3005228975 learned_addr learned my addr 172.21.15.69:0/3005228975 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:02.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.435+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854330873 0 0) 0x7f4a900af940 con 0x7f4a900b4d60 2024-03-25T19:44:02.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.435+0000 7f4a94a0e640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.435+0000 7f4a94a0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.435+0000 7f4a94a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.435+0000 7f4a94a0e640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:44:02.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.435+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c003520 con 0x7f4a900b4d60 2024-03-25T19:44:02.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.441+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744358204 0 0) 0x7f4a900b0a70 con 0x7f4a900b1730 2024-03-25T19:44:02.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.441+0000 7f4a94a0e640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.441+0000 7f4a94a0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.441+0000 7f4a94a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.442 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.441+0000 7f4a94a0e640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:44:02.442 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.441+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a900af940 con 0x7f4a900b1730 2024-03-25T19:44:02.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.444+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426510615 0 0) 0x7f4a900b0d50 con 0x7f4a9014be90 2024-03-25T19:44:02.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.444+0000 7f4a94a0e640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.444+0000 7f4a94a0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.444+0000 7f4a94a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.444+0000 7f4a94a0e640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:44:02.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.444+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a900b0a70 con 0x7f4a9014be90 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2431414344 0 0) 0x7f4a7c003520 con 0x7f4a900b4d60 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 >> v1:172.21.15.69:6790/0 conn(0x7f4a9014be90 legacy=0x7f4a9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 >> v1:172.21.15.169:6789/0 conn(0x7f4a900b1730 legacy=0x7f4a900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 10 monclient: _finish_hunting 0 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 1 monclient: found mon.noname-a 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a900b1030 con 0x7f4a900b4d60 2024-03-25T19:44:02.450 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 10 monclient: _finish_auth 0 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 10 monclient: _check_auth_tickets 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4a8c001690 con 0x7f4a900b4d60 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a94a0e640 10 monclient: dump: 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:02.451 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:02.452 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:02.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a97c74640 5 monclient: authenticate success, global_id 4377 2024-03-25T19:44:02.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.450+0000 7f4a97c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a8c0017f0 con 0x7f4a900b4d60 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a94a0e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a94a0e640 1 -- 172.21.15.69:0/3005228975 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4a8c001c00 con 0x7f4a900b4d60 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a97c74640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a97c74640 4 set_mon_vals no callback set 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a94a0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a94a0e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a94a0e640 10 monclient: dump: 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:02.460 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:02.461 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:02.461 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:02.461 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a97c74640 10 monclient: shutdown 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a97c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a97c74640 1 -- 172.21.15.69:0/3005228975 >> v1:172.21.15.69:6789/0 conn(0x7f4a900b4d60 legacy=0x7f4a9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a97c74640 1 -- 172.21.15.69:0/3005228975 shutdown_connections 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.460+0000 7f4a97c74640 1 -- 172.21.15.69:0/3005228975 >> 172.21.15.69:0/3005228975 conn(0x7f4a900617a0 msgr2=0x7f4a90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.461+0000 7f4a97c74640 1 -- 172.21.15.69:0/3005228975 shutdown_connections 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.461+0000 7f4a97c74640 1 -- 172.21.15.69:0/3005228975 wait complete. 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.461+0000 7f4a97c74640 10 monclient: build_initial_monmap 2024-03-25T19:44:02.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.462+0000 7f4a97c74640 10 monclient: monmap: 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.462+0000 7f4a97c74640 1 librados: starting msgr at 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.462+0000 7f4a97c74640 1 librados: starting objecter 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.462+0000 7f4a97c74640 1 Processor -- start 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.462+0000 7f4a97c74640 1 -- start start 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.462+0000 7f4a97c74640 1 librados: setting wanted keys 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.462+0000 7f4a97c74640 1 librados: calling monclient init 2024-03-25T19:44:02.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.462+0000 7f4a97c74640 10 monclient: init 2024-03-25T19:44:02.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 10 monclient(hunting): picked mon.noname-a con 0x7f4a900b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 10 monclient(hunting): picked mon.noname-c con 0x7f4a900b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 10 monclient(hunting): picked mon.noname-b con 0x7f4a9014be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a90142030 con 0x7f4a900b1730 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9013bdf0 con 0x7f4a9014be90 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a901424c0 con 0x7f4a900b4d60 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:02.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.463+0000 7f4a97c74640 10 monclient(hunting): authenticate will time out at 5054.731802s 2024-03-25T19:44:02.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.476+0000 7f4a95a10640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4a900b1730 0x7f4a90140840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47926/0 (socket says 172.21.15.69:47926) 2024-03-25T19:44:02.476 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.477+0000 7f4a95a10640 1 -- 172.21.15.69:0/3560331383 learned_addr learned my addr 172.21.15.69:0/3560331383 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:02.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.478+0000 7f4a867fc640 10 client.?.objecter ms_handle_connect 0x7f4a900b1730 2024-03-25T19:44:02.478 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.478+0000 7f4a867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:02.484 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.484+0000 7f4a867fc640 10 client.?.objecter ms_handle_connect 0x7f4a9014be90 2024-03-25T19:44:02.484 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.484+0000 7f4a867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:02.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.485+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044725363 0 0) 0x7f4a90142030 con 0x7f4a900b1730 2024-03-25T19:44:02.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.485+0000 7f4a867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.485+0000 7f4a867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.485+0000 7f4a867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.485+0000 7f4a867fc640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:44:02.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.485+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a74004eb0 con 0x7f4a900b1730 2024-03-25T19:44:02.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.488+0000 7f4a867fc640 10 client.?.objecter ms_handle_connect 0x7f4a900b4d60 2024-03-25T19:44:02.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.488+0000 7f4a867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.497+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092315850 0 0) 0x7f4a901424c0 con 0x7f4a900b4d60 2024-03-25T19:44:02.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.497+0000 7f4a867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.497+0000 7f4a867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.497+0000 7f4a867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.497+0000 7f4a867fc640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:44:02.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.497+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a90142030 con 0x7f4a900b4d60 2024-03-25T19:44:02.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.505+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196239333 0 0) 0x7f4a9013bdf0 con 0x7f4a9014be90 2024-03-25T19:44:02.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.505+0000 7f4a867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.505+0000 7f4a867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.505+0000 7f4a867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.505+0000 7f4a867fc640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:44:02.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.505+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a901424c0 con 0x7f4a9014be90 2024-03-25T19:44:02.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.508+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3625577345 0 0) 0x7f4a74004eb0 con 0x7f4a900b1730 2024-03-25T19:44:02.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.508+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a9013bdf0 con 0x7f4a900b1730 2024-03-25T19:44:02.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.508+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4a8c001bc0 con 0x7f4a900b1730 2024-03-25T19:44:02.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.508+0000 7f4a867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.508+0000 7f4a867fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.508+0000 7f4a867fc640 10 monclient(hunting): dump: 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.508+0000 7f4a867fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.516+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 903818591 0 0) 0x7f4a90142030 con 0x7f4a900b4d60 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.516+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a74004eb0 con 0x7f4a900b4d60 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.516+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4a80001970 con 0x7f4a900b4d60 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.516+0000 7f4a867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.516+0000 7f4a867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.516+0000 7f4a867fc640 10 monclient(hunting): dump: 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:02.516 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:02.517 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:02.517 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:02.517 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:02.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 870538163 0 0) 0x7f4a9013bdf0 con 0x7f4a900b1730 2024-03-25T19:44:02.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 >> v1:172.21.15.69:6790/0 conn(0x7f4a900b4d60 legacy=0x7f4a901db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 >> v1:172.21.15.169:6789/0 conn(0x7f4a9014be90 legacy=0x7f4a90142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 1 monclient: found mon.a 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a90142650 con 0x7f4a900b1730 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a97c74640 5 monclient: authenticate success, global_id 4380 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a97c74640 10 monclient: _renew_subs 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a97c74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a901db820 con 0x7f4a900b1730 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.520+0000 7f4a97c74640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:02.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.521+0000 7f4a97c74640 10 monclient: _renew_subs 2024-03-25T19:44:02.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.521+0000 7f4a97c74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:02.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.521+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a901435a0 con 0x7f4a900b1730 2024-03-25T19:44:02.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.521+0000 7f4a97c74640 1 librados: init done 2024-03-25T19:44:02.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.522+0000 7f4a97c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:02.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.522+0000 7f4a97c74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:02.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.522+0000 7f4a97c74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:02.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.522+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a901daf60 con 0x7f4a900b1730 2024-03-25T19:44:02.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.846+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a8c0011f0 con 0x7f4a900b1730 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.846+0000 7f4a867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.846+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4a8c004cb0 con 0x7f4a900b1730 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.846+0000 7f4a867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.846+0000 7f4a867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.846+0000 7f4a867fc640 10 monclient: dump: 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:02.847 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:02.848 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:02.848 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:02.848 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:02.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.846+0000 7f4a87fff640 4 set_mon_vals no callback set 2024-03-25T19:44:02.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.847+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f4a8c01ad00 con 0x7f4a900b1730 2024-03-25T19:44:02.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.848+0000 7f4a9520f640 10 monclient: get_auth_request con 0x7f4a7406ded0 auth_method 0 2024-03-25T19:44:02.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.848+0000 7f4a867fc640 10 client.4380.objecter ms_handle_connect 0x7f4a7406ded0 2024-03-25T19:44:02.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.854+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f4a8c0549d0 con 0x7f4a900b1730 2024-03-25T19:44:02.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.854+0000 7f4a867fc640 10 client.4380.objecter ms_dispatch 0x7f4a90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:02.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.854+0000 7f4a867fc640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:02.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.854+0000 7f4a867fc640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:02.854 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.854+0000 7f4a867fc640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-25T19:44:02.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.854+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a8c050310 con 0x7f4a900b1730 2024-03-25T19:44:02.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.854+0000 7f4a867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:02.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:02.854+0000 7f4a867fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:03.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.074+0000 7f4a97c74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:03.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.074+0000 7f4a97c74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:03.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.074+0000 7f4a97c74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:03.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.074+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a901401f0 con 0x7f4a900b1730 2024-03-25T19:44:03.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.142+0000 7f4a867fc640 1 -- 172.21.15.69:0/3560331383 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f4a8c052410 con 0x7f4a900b1730 2024-03-25T19:44:03.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.142+0000 7f4a867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:03.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.142+0000 7f4a867fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:03.142 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:44:03.142 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 10 librados: watch_flush enter 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 10 librados: watch_flush exit 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 10 client.4380.objecter successfully canceled tick 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 >> v1:172.21.15.169:6800/3072548259 conn(0x7f4a7406ded0 legacy=0x7f4a74070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 10 monclient: shutdown 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 >> v1:172.21.15.69:6789/0 conn(0x7f4a900b1730 legacy=0x7f4a90140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 shutdown_connections 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 >> 172.21.15.69:0/3560331383 conn(0x7f4a900617a0 msgr2=0x7f4a9013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 shutdown_connections 2024-03-25T19:44:03.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 1 -- 172.21.15.69:0/3560331383 wait complete. 2024-03-25T19:44:03.146 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:03.144+0000 7f4a97c74640 1 librados: shutdown 2024-03-25T19:44:03.157 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:09.963 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:10.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.081+0000 7f566570e640 10 monclient: get_monmap_and_config 2024-03-25T19:44:10.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.081+0000 7f566570e640 10 monclient: build_initial_monmap 2024-03-25T19:44:10.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.081+0000 7f566570e640 10 monclient: monmap: 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:44:10.081737+0000 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:44:10.081737+0000 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:44:10.081 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:10.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 1 Processor -- start 2024-03-25T19:44:10.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 1 -- start start 2024-03-25T19:44:10.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 10 monclient: init 2024-03-25T19:44:10.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 10 monclient(hunting): picked mon.noname-b con 0x7f56600a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 10 monclient(hunting): picked mon.noname-a con 0x7f566013f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 10 monclient(hunting): picked mon.noname-c con 0x7f56601430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5660142520 con 0x7f566013f710 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56600b28a0 con 0x7f56601430c0 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56600b2a30 con 0x7f56600a98f0 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:10.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.082+0000 7f566570e640 10 monclient(hunting): authenticate will time out at 5062.350568s 2024-03-25T19:44:10.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.096+0000 7f565e7fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f566013f710 0x7f56601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52844/0 (socket says 172.21.15.69:52844) 2024-03-25T19:44:10.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.097+0000 7f565e7fc640 1 -- 172.21.15.69:0/1894456836 learned_addr learned my addr 172.21.15.69:0/1894456836 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:10.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.111+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176069616 0 0) 0x7f56600b28a0 con 0x7f56601430c0 2024-03-25T19:44:10.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.111+0000 7f565dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.111+0000 7f565dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.111+0000 7f565dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.111+0000 7f565dffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:44:10.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.111+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5648003520 con 0x7f56601430c0 2024-03-25T19:44:10.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.112+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237234332 0 0) 0x7f5660142520 con 0x7f566013f710 2024-03-25T19:44:10.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.112+0000 7f565dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.112 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.112+0000 7f565dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.112 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.112+0000 7f565dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.112 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.112+0000 7f565dffb640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:44:10.112 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.112+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56600b28a0 con 0x7f566013f710 2024-03-25T19:44:10.115 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.116+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118276091 0 0) 0x7f56600b2a30 con 0x7f56600a98f0 2024-03-25T19:44:10.115 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.116+0000 7f565dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.115 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.116+0000 7f565dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.115 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.116+0000 7f565dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.115 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.116+0000 7f565dffb640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:44:10.115 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.116+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5660142520 con 0x7f56600a98f0 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1115584067 0 0) 0x7f5660142520 con 0x7f56600a98f0 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 >> v1:172.21.15.169:6789/0 conn(0x7f56601430c0 legacy=0x7f566014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 >> v1:172.21.15.69:6789/0 conn(0x7f566013f710 legacy=0x7f56601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 1 monclient: found mon.noname-b 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56600b44c0 con 0x7f56600a98f0 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f56500015b0 con 0x7f56600a98f0 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:10.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f565dffb640 10 monclient: dump: 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f566570e640 5 monclient: authenticate success, global_id 4361 2024-03-25T19:44:10.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.128+0000 7f566570e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:10.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.137+0000 7f565dffb640 1 -- 172.21.15.69:0/1894456836 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5650001710 con 0x7f56600a98f0 2024-03-25T19:44:10.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.137+0000 7f565dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:10.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.137+0000 7f566570e640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:10.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.137+0000 7f566570e640 4 set_mon_vals no callback set 2024-03-25T19:44:10.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.137+0000 7f566570e640 10 monclient: shutdown 2024-03-25T19:44:10.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.137+0000 7f566570e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:10.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.137+0000 7f566570e640 1 -- 172.21.15.69:0/1894456836 >> v1:172.21.15.69:6790/0 conn(0x7f56600a98f0 legacy=0x7f56600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.138+0000 7f566570e640 1 -- 172.21.15.69:0/1894456836 shutdown_connections 2024-03-25T19:44:10.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.138+0000 7f566570e640 1 -- 172.21.15.69:0/1894456836 >> 172.21.15.69:0/1894456836 conn(0x7f56600b7ea0 msgr2=0x7f56600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:10.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.138+0000 7f566570e640 1 -- 172.21.15.69:0/1894456836 shutdown_connections 2024-03-25T19:44:10.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.138+0000 7f566570e640 1 -- 172.21.15.69:0/1894456836 wait complete. 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 10 monclient: build_initial_monmap 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 10 monclient: monmap: 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:10.138 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:44:10.139 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:44:10.139 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:10.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 1 librados: starting msgr at 2024-03-25T19:44:10.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 1 librados: starting objecter 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 1 Processor -- start 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 1 -- start start 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 1 librados: setting wanted keys 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 1 librados: calling monclient init 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.139+0000 7f566570e640 10 monclient: init 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 10 monclient(hunting): picked mon.noname-c con 0x7f56600a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:10.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 10 monclient(hunting): picked mon.noname-a con 0x7f566013f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:10.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 10 monclient(hunting): picked mon.noname-b con 0x7f56601430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:10.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56601d8080 con 0x7f566013f710 2024-03-25T19:44:10.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56600b1770 con 0x7f56601430c0 2024-03-25T19:44:10.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56600b09d0 con 0x7f56600a98f0 2024-03-25T19:44:10.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:10.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.140+0000 7f566570e640 10 monclient(hunting): authenticate will time out at 5062.408055s 2024-03-25T19:44:10.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.150+0000 7f565effd640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f56600a98f0 0x7f56601d7740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:47268/0 (socket says 172.21.15.69:47268) 2024-03-25T19:44:10.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.150+0000 7f565effd640 1 -- 172.21.15.69:0/798624933 learned_addr learned my addr 172.21.15.69:0/798624933 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:10.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.151+0000 7f56477fe640 10 client.?.objecter ms_handle_connect 0x7f56600a98f0 2024-03-25T19:44:10.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.151+0000 7f56477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:10.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.153+0000 7f56477fe640 10 client.?.objecter ms_handle_connect 0x7f56601430c0 2024-03-25T19:44:10.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.153+0000 7f56477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:10.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.154+0000 7f56477fe640 10 client.?.objecter ms_handle_connect 0x7f566013f710 2024-03-25T19:44:10.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.154+0000 7f56477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:10.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.155+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203873619 0 0) 0x7f56600b09d0 con 0x7f56600a98f0 2024-03-25T19:44:10.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.155+0000 7f56477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.155+0000 7f56477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.155+0000 7f56477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.155+0000 7f56477fe640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:44:10.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.155+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f564c004eb0 con 0x7f56600a98f0 2024-03-25T19:44:10.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.162+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619293941 0 0) 0x7f56600b1770 con 0x7f56601430c0 2024-03-25T19:44:10.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.162+0000 7f56477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.162+0000 7f56477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.162+0000 7f56477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.162+0000 7f56477fe640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:44:10.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.162+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56600b09d0 con 0x7f56601430c0 2024-03-25T19:44:10.170 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.172+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746496064 0 0) 0x7f56601d8080 con 0x7f566013f710 2024-03-25T19:44:10.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.172+0000 7f56477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.172+0000 7f56477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.172+0000 7f56477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.172+0000 7f56477fe640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:44:10.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.172+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56600b1770 con 0x7f566013f710 2024-03-25T19:44:10.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.180+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 566584188 0 0) 0x7f564c004eb0 con 0x7f56600a98f0 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.181+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56601d8080 con 0x7f56600a98f0 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.181+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5650001d00 con 0x7f56600a98f0 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.181+0000 7f56477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.181+0000 7f56477fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.181+0000 7f56477fe640 10 monclient(hunting): dump: 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:10.180 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:10.181 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:10.181 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:10.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.181+0000 7f56477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:10.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.183+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 743942611 0 0) 0x7f56600b09d0 con 0x7f56601430c0 2024-03-25T19:44:10.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.183+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f564c004eb0 con 0x7f56601430c0 2024-03-25T19:44:10.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.183+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5658001ab0 con 0x7f56601430c0 2024-03-25T19:44:10.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.183+0000 7f56477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.183+0000 7f56477fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.183+0000 7f56477fe640 10 monclient(hunting): dump: 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:10.183 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.186+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3499843206 0 0) 0x7f56600b1770 con 0x7f566013f710 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.186+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56600b09d0 con 0x7f566013f710 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.186+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5654001930 con 0x7f566013f710 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.186+0000 7f56477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.186+0000 7f56477fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.186+0000 7f56477fe640 10 monclient(hunting): dump: 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.186 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.187 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:10.187 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:10.187 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:10.187 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:10.187 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:10.187 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:10.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3105832647 0 0) 0x7f56601d8080 con 0x7f56600a98f0 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 >> v1:172.21.15.169:6789/0 conn(0x7f56601430c0 legacy=0x7f56600acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 >> v1:172.21.15.69:6789/0 conn(0x7f566013f710 legacy=0x7f56600b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 1 monclient: found mon.c 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56600b0c40 con 0x7f56600a98f0 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f56477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 5 monclient: authenticate success, global_id 4364 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 10 monclient: _renew_subs 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56600ad420 con 0x7f56600a98f0 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:10.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 10 monclient: _renew_subs 2024-03-25T19:44:10.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:10.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56600ae960 con 0x7f56600a98f0 2024-03-25T19:44:10.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.192+0000 7f566570e640 1 librados: init done 2024-03-25T19:44:10.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.193+0000 7f566570e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:10.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.193+0000 7f566570e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:10.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.193+0000 7f566570e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:10.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.193+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56600627b0 con 0x7f56600a98f0 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.505+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56500038a0 con 0x7f56600a98f0 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.505+0000 7f56477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.505+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5650003c70 con 0x7f56600a98f0 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.505+0000 7f56477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.505+0000 7f56477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.505+0000 7f56477fe640 10 monclient: dump: 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:10.504 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:10.505 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:10.505 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:10.505 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:10.505 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:10.505 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:10.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.505+0000 7f565d7fa640 4 set_mon_vals no callback set 2024-03-25T19:44:10.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.505+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f5650003e90 con 0x7f56600a98f0 2024-03-25T19:44:10.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.506+0000 7f565e7fc640 10 monclient: get_auth_request con 0x7f564c06e0f0 auth_method 0 2024-03-25T19:44:10.506 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.506+0000 7f56477fe640 10 client.4364.objecter ms_handle_connect 0x7f564c06e0f0 2024-03-25T19:44:10.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.511+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f5650054930 con 0x7f56600a98f0 2024-03-25T19:44:10.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.511+0000 7f56477fe640 10 client.4364.objecter ms_dispatch 0x7f5660000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:10.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.511+0000 7f56477fe640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:10.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.511+0000 7f56477fe640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:10.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.511+0000 7f56477fe640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-25T19:44:10.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.511+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5650050440 con 0x7f56600a98f0 2024-03-25T19:44:10.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.511+0000 7f56477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:10.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.511+0000 7f56477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:10.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.738+0000 7f566570e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:10.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.738+0000 7f566570e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:10.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.739+0000 7f566570e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:10.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.739+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5660069f40 con 0x7f56600a98f0 2024-03-25T19:44:10.818 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.819+0000 7f56477fe640 1 -- 172.21.15.69:0/798624933 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f5650052370 con 0x7f56600a98f0 2024-03-25T19:44:10.818 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.819+0000 7f56477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:10.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.819+0000 7f56477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:10.819 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:44:10.819 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:10.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 10 librados: watch_flush enter 2024-03-25T19:44:10.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 10 librados: watch_flush exit 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 10 client.4364.objecter successfully canceled tick 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 >> v1:172.21.15.169:6800/3072548259 conn(0x7f564c06e0f0 legacy=0x7f564c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 10 monclient: shutdown 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 >> v1:172.21.15.69:6790/0 conn(0x7f56600a98f0 legacy=0x7f56601d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 shutdown_connections 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 >> 172.21.15.69:0/798624933 conn(0x7f56600b7ea0 msgr2=0x7f5660074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 shutdown_connections 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 1 -- 172.21.15.69:0/798624933 wait complete. 2024-03-25T19:44:10.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:10.822+0000 7f566570e640 1 librados: shutdown 2024-03-25T19:44:10.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:17.637 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:17.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.752+0000 7f81bd4a1640 10 monclient: get_monmap_and_config 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.752+0000 7f81bd4a1640 10 monclient: build_initial_monmap 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.752+0000 7f81bd4a1640 10 monclient: monmap: 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:44:17.753819+0000 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:44:17.753819+0000 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:44:17.753 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.753+0000 7f81bd4a1640 1 Processor -- start 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.753+0000 7f81bd4a1640 1 -- start start 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.753+0000 7f81bd4a1640 10 monclient: init 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 10 monclient(hunting): picked mon.noname-c con 0x7f81b80a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 10 monclient(hunting): picked mon.noname-a con 0x7f81b813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 10 monclient(hunting): picked mon.noname-b con 0x7f81b81430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b8142520 con 0x7f81b813f710 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b80b28a0 con 0x7f81b80a98f0 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b80b2a30 con 0x7f81b81430c0 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:17.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.754+0000 7f81bd4a1640 10 monclient(hunting): authenticate will time out at 5070.022966s 2024-03-25T19:44:17.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.767+0000 7f81b6ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f81b80a98f0 0x7f81b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:58712/0 (socket says 172.21.15.69:58712) 2024-03-25T19:44:17.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.767+0000 7f81b6ffd640 1 -- 172.21.15.69:0/3930758695 learned_addr learned my addr 172.21.15.69:0/3930758695 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:17.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.777+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404689102 0 0) 0x7f81b80b28a0 con 0x7f81b80a98f0 2024-03-25T19:44:17.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.777+0000 7f81b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.777+0000 7f81b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.777+0000 7f81b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.777+0000 7f81b5ffb640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:44:17.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.777+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a0003520 con 0x7f81b80a98f0 2024-03-25T19:44:17.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719493747 0 0) 0x7f81b80b2a30 con 0x7f81b81430c0 2024-03-25T19:44:17.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:44:17.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b80b28a0 con 0x7f81b81430c0 2024-03-25T19:44:17.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941731086 0 0) 0x7f81b8142520 con 0x7f81b813f710 2024-03-25T19:44:17.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:44:17.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.797+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b80b2a30 con 0x7f81b813f710 2024-03-25T19:44:17.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2777370898 0 0) 0x7f81a0003520 con 0x7f81b80a98f0 2024-03-25T19:44:17.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 >> v1:172.21.15.69:6790/0 conn(0x7f81b81430c0 legacy=0x7f81b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 >> v1:172.21.15.69:6789/0 conn(0x7f81b813f710 legacy=0x7f81b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b80b44c0 con 0x7f81b80a98f0 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f81a8001690 con 0x7f81b80a98f0 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81b5ffb640 10 monclient: dump: 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:17.801 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81bd4a1640 5 monclient: authenticate success, global_id 4363 2024-03-25T19:44:17.802 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.800+0000 7f81bd4a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.808+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a80017f0 con 0x7f81b80a98f0 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.808+0000 7f81b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.808+0000 7f81b5ffb640 1 -- 172.21.15.69:0/3930758695 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f81a8001c00 con 0x7f81b80a98f0 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.808+0000 7f81bd4a1640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.808+0000 7f81bd4a1640 4 set_mon_vals no callback set 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.808+0000 7f81b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.808+0000 7f81b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.808+0000 7f81b5ffb640 10 monclient: dump: 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:17.808 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:17.809 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:17.809 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:17.809 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.809 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:17.809 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:17.809 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:17.809 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:17.809 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:17.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 10 monclient: shutdown 2024-03-25T19:44:17.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:17.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 1 -- 172.21.15.69:0/3930758695 >> v1:172.21.15.169:6789/0 conn(0x7f81b80a98f0 legacy=0x7f81b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 1 -- 172.21.15.69:0/3930758695 shutdown_connections 2024-03-25T19:44:17.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 1 -- 172.21.15.69:0/3930758695 >> 172.21.15.69:0/3930758695 conn(0x7f81b80b7ea0 msgr2=0x7f81b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:17.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 1 -- 172.21.15.69:0/3930758695 shutdown_connections 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 1 -- 172.21.15.69:0/3930758695 wait complete. 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 10 monclient: build_initial_monmap 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.809+0000 7f81bd4a1640 10 monclient: monmap: 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:17.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.810+0000 7f81bd4a1640 1 librados: starting msgr at 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.810+0000 7f81bd4a1640 1 librados: starting objecter 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.810+0000 7f81bd4a1640 1 Processor -- start 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.810+0000 7f81bd4a1640 1 -- start start 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.810+0000 7f81bd4a1640 1 librados: setting wanted keys 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.810+0000 7f81bd4a1640 1 librados: calling monclient init 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.810+0000 7f81bd4a1640 10 monclient: init 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 10 monclient(hunting): picked mon.noname-a con 0x7f81b80a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 10 monclient(hunting): picked mon.noname-c con 0x7f81b813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 10 monclient(hunting): picked mon.noname-b con 0x7f81b81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b81d7380 con 0x7f81b80a98f0 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b8139840 con 0x7f81b81430c0 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b81d8200 con 0x7f81b813f710 2024-03-25T19:44:17.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:17.814 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.811+0000 7f81bd4a1640 10 monclient(hunting): authenticate will time out at 5070.079794s 2024-03-25T19:44:17.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.823+0000 7f81b77fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f81b81430c0 0x7f81b81daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:58740/0 (socket says 172.21.15.69:58740) 2024-03-25T19:44:17.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.823+0000 7f81b77fe640 1 -- 172.21.15.69:0/740318511 learned_addr learned my addr 172.21.15.69:0/740318511 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:17.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.825+0000 7f819f7fe640 10 client.?.objecter ms_handle_connect 0x7f81b81430c0 2024-03-25T19:44:17.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.825+0000 7f819f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:17.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.829+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232863508 0 0) 0x7f81b8139840 con 0x7f81b81430c0 2024-03-25T19:44:17.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.829+0000 7f819f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.829+0000 7f819f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.829+0000 7f819f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.829+0000 7f819f7fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:44:17.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.829+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a4004eb0 con 0x7f81b81430c0 2024-03-25T19:44:17.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.838+0000 7f819f7fe640 10 client.?.objecter ms_handle_connect 0x7f81b813f710 2024-03-25T19:44:17.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.838+0000 7f819f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.839+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2015249450 0 0) 0x7f81a4004eb0 con 0x7f81b81430c0 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.839+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81b8139840 con 0x7f81b81430c0 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.839+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f81b0001c90 con 0x7f81b81430c0 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.839+0000 7f819f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.839+0000 7f819f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.839+0000 7f819f7fe640 10 monclient(hunting): dump: 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:17.839 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.840 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:17.840 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:17.840 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:17.840 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:17.840 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:17.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.839+0000 7f819f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:17.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.840+0000 7f819f7fe640 10 client.?.objecter ms_handle_connect 0x7f81b80a98f0 2024-03-25T19:44:17.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.840+0000 7f819f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:17.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3285597826 0 0) 0x7f81b8139840 con 0x7f81b81430c0 2024-03-25T19:44:17.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 >> v1:172.21.15.69:6790/0 conn(0x7f81b813f710 legacy=0x7f81b81da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 >> v1:172.21.15.69:6789/0 conn(0x7f81b80a98f0 legacy=0x7f81b81d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 1 monclient: found mon.b 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b81d75d0 con 0x7f81b81430c0 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f819f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 10 monclient: _renew_subs 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b81db6a0 con 0x7f81b81430c0 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:17.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 10 monclient: _renew_subs 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b8137910 con 0x7f81b81430c0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.855+0000 7f81bd4a1640 1 librados: init done 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.856+0000 7f81bd4a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.856+0000 7f81bd4a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.856+0000 7f81bd4a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.856+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b80627b0 con 0x7f81b81430c0 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.880+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81b0003970 con 0x7f81b81430c0 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.880+0000 7f819f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.880+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f81b0001df0 con 0x7f81b81430c0 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.880+0000 7f819f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.880+0000 7f819f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.880+0000 7f819f7fe640 10 monclient: dump: 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:17.881 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:17.882 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:17.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.880+0000 7f81b57fa640 4 set_mon_vals no callback set 2024-03-25T19:44:17.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.881+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f81b0003e00 con 0x7f81b81430c0 2024-03-25T19:44:17.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.882+0000 7f81b6ffd640 10 monclient: get_auth_request con 0x7f81a406dd90 auth_method 0 2024-03-25T19:44:17.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.882+0000 7f819f7fe640 10 client.4366.objecter ms_handle_connect 0x7f81a406dd90 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.885+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f81b0054b40 con 0x7f81b81430c0 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.885+0000 7f819f7fe640 10 client.4366.objecter ms_dispatch 0x7f81b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.885+0000 7f819f7fe640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.885+0000 7f819f7fe640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:17.885+0000 7f819f7fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:44:18.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.153+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81b0050650 con 0x7f81b81430c0 2024-03-25T19:44:18.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.153+0000 7f819f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:18.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.153+0000 7f819f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:18.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.401+0000 7f81bd4a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:18.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.401+0000 7f81bd4a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:18.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.401+0000 7f81bd4a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:18.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.401+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81b81d7760 con 0x7f81b81430c0 2024-03-25T19:44:18.436 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.435+0000 7f819f7fe640 1 -- 172.21.15.69:0/740318511 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f81b0052580 con 0x7f81b81430c0 2024-03-25T19:44:18.436 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.435+0000 7f819f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:18.436 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.435+0000 7f819f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:18.436 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:44:18.436 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:18.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 10 librados: watch_flush enter 2024-03-25T19:44:18.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 10 librados: watch_flush exit 2024-03-25T19:44:18.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:44:18.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:44:18.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 >> v1:172.21.15.169:6800/3072548259 conn(0x7f81a406dd90 legacy=0x7f81a4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:18.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 10 monclient: shutdown 2024-03-25T19:44:18.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:18.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 >> v1:172.21.15.169:6789/0 conn(0x7f81b81430c0 legacy=0x7f81b81daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:18.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 shutdown_connections 2024-03-25T19:44:18.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 >> 172.21.15.69:0/740318511 conn(0x7f81b80b7ea0 msgr2=0x7f81b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:18.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 shutdown_connections 2024-03-25T19:44:18.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 1 -- 172.21.15.69:0/740318511 wait complete. 2024-03-25T19:44:18.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:18.438+0000 7f81bd4a1640 1 librados: shutdown 2024-03-25T19:44:18.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:21.953 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:59.917 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:21.964 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:59.928 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:21.996 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:59.960 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:22.012 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:59.966 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:22.015 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:59.970 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:22.020 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:59.984 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:22.022 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:59.986 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:22.044 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:00.008 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:22.055 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:00.009 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:25.257 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:25.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.372+0000 7f0cebf3b640 10 monclient: get_monmap_and_config 2024-03-25T19:44:25.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.372+0000 7f0cebf3b640 10 monclient: build_initial_monmap 2024-03-25T19:44:25.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.372+0000 7f0cebf3b640 10 monclient: monmap: 2024-03-25T19:44:25.373 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:25.373 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:25.373 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:44:25.373812+0000 2024-03-25T19:44:25.373 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:44:25.373812+0000 2024-03-25T19:44:25.373 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:25.374 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:25.374 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:25.374 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:44:25.374 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:44:25.374 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:25.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.373+0000 7f0cebf3b640 1 Processor -- start 2024-03-25T19:44:25.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.373+0000 7f0cebf3b640 1 -- start start 2024-03-25T19:44:25.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.373+0000 7f0cebf3b640 10 monclient: init 2024-03-25T19:44:25.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:25.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:25.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce40a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:25.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:25.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce41430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:25.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce4142520 con 0x7f0ce40a98f0 2024-03-25T19:44:25.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40b28a0 con 0x7f0ce413f710 2024-03-25T19:44:25.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40b2a30 con 0x7f0ce41430c0 2024-03-25T19:44:25.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:25.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.374+0000 7f0cebf3b640 10 monclient(hunting): authenticate will time out at 5077.643004s 2024-03-25T19:44:25.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.382+0000 7f0cea4d8640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f0ce41430c0 0x7f0ce414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:43146/0 (socket says 172.21.15.69:43146) 2024-03-25T19:44:25.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.382+0000 7f0cea4d8640 1 -- 172.21.15.69:0/446010579 learned_addr learned my addr 172.21.15.69:0/446010579 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:25.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.393+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530051052 0 0) 0x7f0ce40b2a30 con 0x7f0ce41430c0 2024-03-25T19:44:25.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.393+0000 7f0ce8cd5640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.393+0000 7f0ce8cd5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.393+0000 7f0ce8cd5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.393+0000 7f0ce8cd5640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:44:25.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.393+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd0003520 con 0x7f0ce41430c0 2024-03-25T19:44:25.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474212130 0 0) 0x7f0cd0003520 con 0x7f0ce41430c0 2024-03-25T19:44:25.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 >> v1:172.21.15.169:6789/0 conn(0x7f0ce413f710 legacy=0x7f0ce41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 >> v1:172.21.15.69:6789/0 conn(0x7f0ce40a98f0 legacy=0x7f0ce40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:44:25.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 10 monclient: _finish_hunting 0 2024-03-25T19:44:25.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 1 monclient: found mon.noname-b 2024-03-25T19:44:25.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:25.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:44:25.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40b44c0 con 0x7f0ce41430c0 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 10 monclient: _finish_auth 0 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 10 monclient: _check_auth_tickets 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0cdc001a10 con 0x7f0ce41430c0 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0cebf3b640 5 monclient: authenticate success, global_id 4373 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0cebf3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0ce8cd5640 10 monclient: dump: 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:25.404 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.405 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.405 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.405 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:25.405 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:25.405 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:25.405 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:25.405 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:25.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.402+0000 7f0cebf3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:25.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.413+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cdc001be0 con 0x7f0ce41430c0 2024-03-25T19:44:25.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.413+0000 7f0ce8cd5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:25.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.413+0000 7f0ce8cd5640 1 -- 172.21.15.69:0/446010579 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0cdc002030 con 0x7f0ce41430c0 2024-03-25T19:44:25.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.413+0000 7f0ce8cd5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.413+0000 7f0ce8cd5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:25.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.413+0000 7f0ce8cd5640 10 monclient: dump: 2024-03-25T19:44:25.413 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.413+0000 7f0cebf3b640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:25.414 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.413+0000 7f0cebf3b640 4 set_mon_vals no callback set 2024-03-25T19:44:25.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.414+0000 7f0cebf3b640 10 monclient: shutdown 2024-03-25T19:44:25.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.414+0000 7f0cebf3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:25.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.414+0000 7f0cebf3b640 1 -- 172.21.15.69:0/446010579 >> v1:172.21.15.69:6790/0 conn(0x7f0ce41430c0 legacy=0x7f0ce414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.414+0000 7f0cebf3b640 1 -- 172.21.15.69:0/446010579 shutdown_connections 2024-03-25T19:44:25.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.414+0000 7f0cebf3b640 1 -- 172.21.15.69:0/446010579 >> 172.21.15.69:0/446010579 conn(0x7f0ce40b7ea0 msgr2=0x7f0ce40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.414+0000 7f0cebf3b640 1 -- 172.21.15.69:0/446010579 shutdown_connections 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.414+0000 7f0cebf3b640 1 -- 172.21.15.69:0/446010579 wait complete. 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.415+0000 7f0cebf3b640 10 monclient: build_initial_monmap 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.415+0000 7f0cebf3b640 10 monclient: monmap: 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:25.418 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.415+0000 7f0cebf3b640 1 librados: starting msgr at 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.415+0000 7f0cebf3b640 1 librados: starting objecter 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.415+0000 7f0cebf3b640 1 Processor -- start 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.416+0000 7f0cebf3b640 1 -- start start 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.416+0000 7f0cebf3b640 1 librados: setting wanted keys 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.416+0000 7f0cebf3b640 1 librados: calling monclient init 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.416+0000 7f0cebf3b640 10 monclient: init 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.416+0000 7f0cebf3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce40a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:25.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:25.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce41430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:25.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce4137ce0 con 0x7f0ce41430c0 2024-03-25T19:44:25.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce4137980 con 0x7f0ce413f710 2024-03-25T19:44:25.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce4139620 con 0x7f0ce40a98f0 2024-03-25T19:44:25.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:25.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.417+0000 7f0cebf3b640 10 monclient(hunting): authenticate will time out at 5077.686393s 2024-03-25T19:44:25.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.433+0000 7f0cea4d8640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0ce41430c0 0x7f0ce41343d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47734/0 (socket says 172.21.15.69:47734) 2024-03-25T19:44:25.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.433+0000 7f0cea4d8640 1 -- 172.21.15.69:0/3156074499 learned_addr learned my addr 172.21.15.69:0/3156074499 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:25.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.437+0000 7f0cda7fc640 10 client.?.objecter ms_handle_connect 0x7f0ce40a98f0 2024-03-25T19:44:25.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.437+0000 7f0cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:25.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.437+0000 7f0cda7fc640 10 client.?.objecter ms_handle_connect 0x7f0ce413f710 2024-03-25T19:44:25.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.437+0000 7f0cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.440+0000 7f0cda7fc640 10 client.?.objecter ms_handle_connect 0x7f0ce41430c0 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.440+0000 7f0cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:25.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.444+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106212112 0 0) 0x7f0ce4139620 con 0x7f0ce40a98f0 2024-03-25T19:44:25.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.444+0000 7f0cda7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.444+0000 7f0cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.444+0000 7f0cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.444+0000 7f0cda7fc640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:44:25.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.444+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc8004eb0 con 0x7f0ce40a98f0 2024-03-25T19:44:25.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.451+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670060827 0 0) 0x7f0ce4137980 con 0x7f0ce413f710 2024-03-25T19:44:25.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.451+0000 7f0cda7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.451+0000 7f0cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.451+0000 7f0cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.451+0000 7f0cda7fc640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:44:25.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.451+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce4139620 con 0x7f0ce413f710 2024-03-25T19:44:25.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.458+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843848681 0 0) 0x7f0ce4137ce0 con 0x7f0ce41430c0 2024-03-25T19:44:25.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.458+0000 7f0cda7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.458+0000 7f0cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.458+0000 7f0cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.458+0000 7f0cda7fc640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:44:25.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.458+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce4137980 con 0x7f0ce41430c0 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.461+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1178334618 0 0) 0x7f0cc8004eb0 con 0x7f0ce40a98f0 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.461+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce4137ce0 con 0x7f0ce40a98f0 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.461+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0ce0001870 con 0x7f0ce40a98f0 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.461+0000 7f0cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.461+0000 7f0cda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.461+0000 7f0cda7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:25.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.461+0000 7f0cda7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:25.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.465+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2904808327 0 0) 0x7f0ce4139620 con 0x7f0ce413f710 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.465+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cc8004eb0 con 0x7f0ce413f710 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.465+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0cd40018b0 con 0x7f0ce413f710 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.465+0000 7f0cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.465+0000 7f0cda7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.465+0000 7f0cda7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:25.467 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:25.467 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:25.467 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:25.467 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.494+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2251082337 0 0) 0x7f0ce4137980 con 0x7f0ce41430c0 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.494+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce4139620 con 0x7f0ce41430c0 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.494+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0cdc001ef0 con 0x7f0ce41430c0 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.494+0000 7f0cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.494+0000 7f0cda7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.494+0000 7f0cda7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:25.495 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:25.496 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:25.496 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 492857225 0 0) 0x7f0ce4139620 con 0x7f0ce41430c0 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 >> v1:172.21.15.69:6790/0 conn(0x7f0ce40a98f0 legacy=0x7f0ce41daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 >> v1:172.21.15.169:6789/0 conn(0x7f0ce413f710 legacy=0x7f0ce4133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 1 monclient: found mon.a 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce4134ae0 con 0x7f0ce41430c0 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:25.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 5 monclient: authenticate success, global_id 4395 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 10 monclient: _renew_subs 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce4138cf0 con 0x7f0ce41430c0 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 10 monclient: _renew_subs 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce40abac0 con 0x7f0ce41430c0 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.572+0000 7f0cebf3b640 1 librados: init done 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.573+0000 7f0cebf3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:25.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.573+0000 7f0cebf3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:25.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.573+0000 7f0cebf3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:25.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.573+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce40b3ef0 con 0x7f0ce41430c0 2024-03-25T19:44:25.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.877+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cdc002070 con 0x7f0ce41430c0 2024-03-25T19:44:25.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.877+0000 7f0cda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:25.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.877+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0cdc001380 con 0x7f0ce41430c0 2024-03-25T19:44:25.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.877+0000 7f0cda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.877+0000 7f0cda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:25.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.877+0000 7f0cda7fc640 10 monclient: dump: 2024-03-25T19:44:25.877 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:25.877 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:25.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.877+0000 7f0cdbfff640 4 set_mon_vals no callback set 2024-03-25T19:44:25.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.877+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f0cdc01ae20 con 0x7f0ce41430c0 2024-03-25T19:44:25.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.878+0000 7f0ce9cd7640 10 monclient: get_auth_request con 0x7f0cc806e0c0 auth_method 0 2024-03-25T19:44:25.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.878+0000 7f0cda7fc640 10 client.4395.objecter ms_handle_connect 0x7f0cc806e0c0 2024-03-25T19:44:25.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.884+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f0cdc054af0 con 0x7f0ce41430c0 2024-03-25T19:44:25.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.884+0000 7f0cda7fc640 10 client.4395.objecter ms_dispatch 0x7f0ce4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:25.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.884+0000 7f0cda7fc640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:25.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.884+0000 7f0cda7fc640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:25.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.884+0000 7f0cda7fc640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-25T19:44:25.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.884+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cdc050430 con 0x7f0ce41430c0 2024-03-25T19:44:25.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.884+0000 7f0cda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:25.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:25.884+0000 7f0cda7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:26.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.140+0000 7f0cebf3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:26.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.140+0000 7f0cebf3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:26.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.140+0000 7f0cebf3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:26.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.140+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ce40627b0 con 0x7f0ce41430c0 2024-03-25T19:44:26.199 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.198+0000 7f0cda7fc640 1 -- 172.21.15.69:0/3156074499 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f0cdc052530 con 0x7f0ce41430c0 2024-03-25T19:44:26.199 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.198+0000 7f0cda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:26.199 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.198+0000 7f0cda7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:26.199 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:44:26.199 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:26.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 10 librados: watch_flush enter 2024-03-25T19:44:26.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 10 librados: watch_flush exit 2024-03-25T19:44:26.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 10 client.4395.objecter successfully canceled tick 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 >> v1:172.21.15.169:6800/3072548259 conn(0x7f0cc806e0c0 legacy=0x7f0cc8070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 10 monclient: shutdown 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 >> v1:172.21.15.69:6789/0 conn(0x7f0ce41430c0 legacy=0x7f0ce41343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 shutdown_connections 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 >> 172.21.15.69:0/3156074499 conn(0x7f0ce40b7ea0 msgr2=0x7f0ce413c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 shutdown_connections 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 1 -- 172.21.15.69:0/3156074499 wait complete. 2024-03-25T19:44:26.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:26.201+0000 7f0cebf3b640 1 librados: shutdown 2024-03-25T19:44:26.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:33.019 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:33.132 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.132+0000 7f89c071c640 10 monclient: get_monmap_and_config 2024-03-25T19:44:33.132 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.132+0000 7f89c071c640 10 monclient: build_initial_monmap 2024-03-25T19:44:33.132 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.132+0000 7f89c071c640 10 monclient: monmap: 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:44:33.133559+0000 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:44:33.133559+0000 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:44:33.133 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:33.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.133+0000 7f89c071c640 1 Processor -- start 2024-03-25T19:44:33.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.133+0000 7f89c071c640 1 -- start start 2024-03-25T19:44:33.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.133+0000 7f89c071c640 10 monclient: init 2024-03-25T19:44:33.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.133+0000 7f89c071c640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:33.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:33.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 10 monclient(hunting): picked mon.noname-b con 0x7f89b80a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:33.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 10 monclient(hunting): picked mon.noname-a con 0x7f89b813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:33.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 10 monclient(hunting): picked mon.noname-c con 0x7f89b81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8142520 con 0x7f89b813f710 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b80b28a0 con 0x7f89b81430c0 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b80b2a30 con 0x7f89b80a98f0 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.134+0000 7f89c071c640 10 monclient(hunting): authenticate will time out at 5085.402849s 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.135+0000 7f89be4b8640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f89b80a98f0 0x7f89b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:38824/0 (socket says 172.21.15.69:38824) 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.135+0000 7f89be4b8640 1 -- 172.21.15.69:0/100109983 learned_addr learned my addr 172.21.15.69:0/100109983 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:33.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.142+0000 7f89bd4b6640 1 -- 172.21.15.69:0/100109983 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632986773 0 0) 0x7f89b80b2a30 con 0x7f89b80a98f0 2024-03-25T19:44:33.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.142+0000 7f89bd4b6640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.142+0000 7f89bd4b6640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.142+0000 7f89bd4b6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.142+0000 7f89bd4b6640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:44:33.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.142+0000 7f89bd4b6640 1 -- 172.21.15.69:0/100109983 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a4003520 con 0x7f89b80a98f0 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 1 -- 172.21.15.69:0/100109983 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041212990 0 0) 0x7f89a4003520 con 0x7f89b80a98f0 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 1 -- 172.21.15.69:0/100109983 >> v1:172.21.15.169:6789/0 conn(0x7f89b81430c0 legacy=0x7f89b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 1 -- 172.21.15.69:0/100109983 >> v1:172.21.15.69:6789/0 conn(0x7f89b813f710 legacy=0x7f89b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 10 monclient: _finish_hunting 0 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 1 monclient: found mon.noname-b 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 1 -- 172.21.15.69:0/100109983 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b80b44c0 con 0x7f89b80a98f0 2024-03-25T19:44:33.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 10 monclient: _finish_auth 0 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 10 monclient: _check_auth_tickets 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 1 -- 172.21.15.69:0/100109983 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f89b4001690 con 0x7f89b80a98f0 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89bd4b6640 10 monclient: dump: 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:33.151 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:33.151 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:33.151 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:33.151 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:33.151 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:33.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89c071c640 5 monclient: authenticate success, global_id 4379 2024-03-25T19:44:33.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.148+0000 7f89c071c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:33.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.151+0000 7f89bd4b6640 1 -- 172.21.15.69:0/100109983 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89b4001860 con 0x7f89b80a98f0 2024-03-25T19:44:33.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.151+0000 7f89bd4b6640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:33.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.151+0000 7f89c071c640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:33.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.151+0000 7f89c071c640 4 set_mon_vals no callback set 2024-03-25T19:44:33.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.152+0000 7f89c071c640 10 monclient: shutdown 2024-03-25T19:44:33.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.152+0000 7f89c071c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:33.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.152+0000 7f89c071c640 1 -- 172.21.15.69:0/100109983 >> v1:172.21.15.69:6790/0 conn(0x7f89b80a98f0 legacy=0x7f89b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.152+0000 7f89c071c640 1 -- 172.21.15.69:0/100109983 shutdown_connections 2024-03-25T19:44:33.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.152+0000 7f89c071c640 1 -- 172.21.15.69:0/100109983 >> 172.21.15.69:0/100109983 conn(0x7f89b80b7ea0 msgr2=0x7f89b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:33.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.152+0000 7f89c071c640 1 -- 172.21.15.69:0/100109983 shutdown_connections 2024-03-25T19:44:33.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.152+0000 7f89c071c640 1 -- 172.21.15.69:0/100109983 wait complete. 2024-03-25T19:44:33.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.153+0000 7f89c071c640 10 monclient: build_initial_monmap 2024-03-25T19:44:33.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.153+0000 7f89c071c640 10 monclient: monmap: 2024-03-25T19:44:33.154 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:33.154 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:33.154 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:44:33.154 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:44:33.154 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:33.154 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.153+0000 7f89c071c640 1 librados: starting msgr at 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.153+0000 7f89c071c640 1 librados: starting objecter 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.154+0000 7f89c071c640 1 Processor -- start 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.154+0000 7f89c071c640 1 -- start start 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.154+0000 7f89c071c640 1 librados: setting wanted keys 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.154+0000 7f89c071c640 1 librados: calling monclient init 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.154+0000 7f89c071c640 10 monclient: init 2024-03-25T19:44:33.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.154+0000 7f89c071c640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:33.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.154+0000 7f89c071c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:33.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.154+0000 7f89c071c640 10 monclient(hunting): picked mon.noname-a con 0x7f89b80a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:33.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.155+0000 7f89c071c640 10 monclient(hunting): picked mon.noname-b con 0x7f89b813f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:33.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.155+0000 7f89c071c640 10 monclient(hunting): picked mon.noname-c con 0x7f89b81430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:33.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.155+0000 7f89c071c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8136460 con 0x7f89b80a98f0 2024-03-25T19:44:33.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.155+0000 7f89c071c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b80aba10 con 0x7f89b813f710 2024-03-25T19:44:33.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.155+0000 7f89c071c640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b81da880 con 0x7f89b81430c0 2024-03-25T19:44:33.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.155+0000 7f89c071c640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:33.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.155+0000 7f89c071c640 10 monclient(hunting): authenticate will time out at 5085.423710s 2024-03-25T19:44:33.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.169+0000 7f89becb9640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f89b81430c0 0x7f89b81d78e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:38836/0 (socket says 172.21.15.69:38836) 2024-03-25T19:44:33.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.169+0000 7f89becb9640 1 -- 172.21.15.69:0/1130857797 learned_addr learned my addr 172.21.15.69:0/1130857797 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.169+0000 7f89aeffd640 10 client.?.objecter ms_handle_connect 0x7f89b81430c0 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.169+0000 7f89aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:33.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.172+0000 7f89aeffd640 10 client.?.objecter ms_handle_connect 0x7f89b80a98f0 2024-03-25T19:44:33.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.172+0000 7f89aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:33.175 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.175+0000 7f89aeffd640 10 client.?.objecter ms_handle_connect 0x7f89b813f710 2024-03-25T19:44:33.175 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.175+0000 7f89aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:33.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.184+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35927000 0 0) 0x7f89b80aba10 con 0x7f89b813f710 2024-03-25T19:44:33.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.184+0000 7f89aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.184+0000 7f89aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.184+0000 7f89aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.184+0000 7f89aeffd640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:44:33.185 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.184+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f899c004eb0 con 0x7f89b813f710 2024-03-25T19:44:33.185 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291391890 0 0) 0x7f89b81da880 con 0x7f89b81430c0 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b80aba10 con 0x7f89b81430c0 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227411475 0 0) 0x7f89b8136460 con 0x7f89b80a98f0 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:44:33.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.185+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b81da880 con 0x7f89b80a98f0 2024-03-25T19:44:33.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.189+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1474032381 0 0) 0x7f899c004eb0 con 0x7f89b813f710 2024-03-25T19:44:33.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.189+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89b8136460 con 0x7f89b813f710 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.189+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f89a8001870 con 0x7f89b813f710 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.189+0000 7f89aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.189+0000 7f89aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.189+0000 7f89aeffd640 10 monclient(hunting): dump: 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:33.190 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:33.191 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.189+0000 7f89aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:33.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 828884783 0 0) 0x7f89b8136460 con 0x7f89b813f710 2024-03-25T19:44:33.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 >> v1:172.21.15.69:6790/0 conn(0x7f89b81430c0 legacy=0x7f89b81d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 >> v1:172.21.15.69:6789/0 conn(0x7f89b80a98f0 legacy=0x7f89b8138190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:33.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 1 monclient: found mon.b 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b81376d0 con 0x7f89b813f710 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 5 monclient: authenticate success, global_id 4375 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 10 monclient: _renew_subs 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b81d7380 con 0x7f89b813f710 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:33.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 10 monclient: _renew_subs 2024-03-25T19:44:33.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:33.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b8138a80 con 0x7f89b813f710 2024-03-25T19:44:33.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.210+0000 7f89c071c640 1 librados: init done 2024-03-25T19:44:33.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.211+0000 7f89c071c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:33.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.211+0000 7f89c071c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:33.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.211+0000 7f89c071c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:33.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.211+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b80627b0 con 0x7f89b813f710 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.510+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89a80035f0 con 0x7f89b813f710 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.510+0000 7f89aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.510+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f89a8001d80 con 0x7f89b813f710 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.510+0000 7f89aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.511+0000 7f89aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.511+0000 7f89aeffd640 10 monclient: dump: 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:33.511 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:33.512 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:33.512 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:33.512 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:33.512 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:33.512 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.510+0000 7f89bccb5640 4 set_mon_vals no callback set 2024-03-25T19:44:33.512 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.512+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f89a80197f0 con 0x7f89b813f710 2024-03-25T19:44:33.512 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.512+0000 7f89be4b8640 10 monclient: get_auth_request con 0x7f899c06dd50 auth_method 0 2024-03-25T19:44:33.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.512+0000 7f89aeffd640 10 client.4375.objecter ms_handle_connect 0x7f899c06dd50 2024-03-25T19:44:33.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.516+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f89a80535d0 con 0x7f89b813f710 2024-03-25T19:44:33.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.516+0000 7f89aeffd640 10 client.4375.objecter ms_dispatch 0x7f89b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:33.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.516+0000 7f89aeffd640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:33.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.516+0000 7f89aeffd640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:33.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.516+0000 7f89aeffd640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T19:44:33.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.516+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89a804ef10 con 0x7f89b813f710 2024-03-25T19:44:33.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.516+0000 7f89aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:33.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.516+0000 7f89aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:33.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.746+0000 7f89c071c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:33.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.746+0000 7f89c071c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:33.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.746+0000 7f89c071c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:33.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.746+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89b813b430 con 0x7f89b813f710 2024-03-25T19:44:33.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.777+0000 7f89aeffd640 1 -- 172.21.15.69:0/1130857797 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f89a8051010 con 0x7f89b813f710 2024-03-25T19:44:33.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.777+0000 7f89aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:33.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.777+0000 7f89aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:33.777 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:44:33.778 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:33.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 10 librados: watch_flush enter 2024-03-25T19:44:33.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 10 librados: watch_flush exit 2024-03-25T19:44:33.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T19:44:33.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 10 client.4375.objecter successfully canceled tick 2024-03-25T19:44:33.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 >> v1:172.21.15.169:6800/3072548259 conn(0x7f899c06dd50 legacy=0x7f899c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 10 monclient: shutdown 2024-03-25T19:44:33.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:33.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 >> v1:172.21.15.169:6789/0 conn(0x7f89b813f710 legacy=0x7f89b81dada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 shutdown_connections 2024-03-25T19:44:33.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.779+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 >> 172.21.15.69:0/1130857797 conn(0x7f89b80b7ea0 msgr2=0x7f89b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:33.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.780+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 shutdown_connections 2024-03-25T19:44:33.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.780+0000 7f89c071c640 1 -- 172.21.15.69:0/1130857797 wait complete. 2024-03-25T19:44:33.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:33.780+0000 7f89c071c640 1 librados: shutdown 2024-03-25T19:44:33.792 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:40.598 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:40.700 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.700+0000 7f761049a640 10 monclient: get_monmap_and_config 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.700+0000 7f761049a640 10 monclient: build_initial_monmap 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.700+0000 7f761049a640 10 monclient: monmap: 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:44:40.701852+0000 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:44:40.701852+0000 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:44:40.701 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:40.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.701+0000 7f761049a640 1 Processor -- start 2024-03-25T19:44:40.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.701+0000 7f761049a640 1 -- start start 2024-03-25T19:44:40.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.701+0000 7f761049a640 10 monclient: init 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.701+0000 7f761049a640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.701+0000 7f761049a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f761049a640 10 monclient(hunting): picked mon.noname-a con 0x7f76080a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f761049a640 10 monclient(hunting): picked mon.noname-b con 0x7f760813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f761049a640 10 monclient(hunting): picked mon.noname-c con 0x7f76081430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f761049a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7608142520 con 0x7f76080a98f0 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f761049a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080b28a0 con 0x7f76081430c0 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f761049a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080b2a30 con 0x7f760813f710 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f761049a640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f761049a640 10 monclient(hunting): authenticate will time out at 5092.970738s 2024-03-25T19:44:40.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f760da35640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f760813f710 0x7f76081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:34090/0 (socket says 172.21.15.69:34090) 2024-03-25T19:44:40.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.702+0000 7f760da35640 1 -- 172.21.15.69:0/2472291246 learned_addr learned my addr 172.21.15.69:0/2472291246 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:40.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.739+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349821504 0 0) 0x7f7608142520 con 0x7f76080a98f0 2024-03-25T19:44:40.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.739+0000 7f760d234640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.739+0000 7f760d234640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.739+0000 7f760d234640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.739+0000 7f760d234640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:44:40.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.739+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f4003520 con 0x7f76080a98f0 2024-03-25T19:44:40.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3768199413 0 0) 0x7f75f4003520 con 0x7f76080a98f0 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 >> v1:172.21.15.69:6790/0 conn(0x7f760813f710 legacy=0x7f76081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 >> v1:172.21.15.169:6789/0 conn(0x7f76081430c0 legacy=0x7f760814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 10 monclient: _finish_hunting 0 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 1 monclient: found mon.noname-a 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080b44c0 con 0x7f76080a98f0 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 10 monclient: _finish_auth 0 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 10 monclient: _check_auth_tickets 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:40.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7604001690 con 0x7f76080a98f0 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f760d234640 10 monclient: dump: 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:40.753 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:40.754 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:40.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f761049a640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:44:40.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.751+0000 7f761049a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.780+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76040017f0 con 0x7f76080a98f0 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.780+0000 7f760d234640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.780+0000 7f760d234640 1 -- 172.21.15.69:0/2472291246 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7604001c00 con 0x7f76080a98f0 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.780+0000 7f760d234640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.780+0000 7f760d234640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.780+0000 7f760d234640 10 monclient: dump: 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:40.781 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.782 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:40.782 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:40.782 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:40.782 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:40.782 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:40.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.781+0000 7f761049a640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:40.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.781+0000 7f761049a640 4 set_mon_vals no callback set 2024-03-25T19:44:40.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.781+0000 7f761049a640 10 monclient: shutdown 2024-03-25T19:44:40.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.781+0000 7f761049a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:40.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.781+0000 7f761049a640 1 -- 172.21.15.69:0/2472291246 >> v1:172.21.15.69:6789/0 conn(0x7f76080a98f0 legacy=0x7f76080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.782+0000 7f761049a640 1 -- 172.21.15.69:0/2472291246 shutdown_connections 2024-03-25T19:44:40.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.782+0000 7f761049a640 1 -- 172.21.15.69:0/2472291246 >> 172.21.15.69:0/2472291246 conn(0x7f76080b7ea0 msgr2=0x7f76080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:40.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.782+0000 7f761049a640 1 -- 172.21.15.69:0/2472291246 shutdown_connections 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.782+0000 7f761049a640 1 -- 172.21.15.69:0/2472291246 wait complete. 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.782+0000 7f761049a640 10 monclient: build_initial_monmap 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.782+0000 7f761049a640 10 monclient: monmap: 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:40.785 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.783+0000 7f761049a640 1 librados: starting msgr at 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.783+0000 7f761049a640 1 librados: starting objecter 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.783+0000 7f761049a640 1 Processor -- start 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.783+0000 7f761049a640 1 -- start start 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.783+0000 7f761049a640 1 librados: setting wanted keys 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.783+0000 7f761049a640 1 librados: calling monclient init 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.783+0000 7f761049a640 10 monclient: init 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 10 monclient(hunting): picked mon.noname-b con 0x7f76080a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 10 monclient(hunting): picked mon.noname-a con 0x7f760813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 10 monclient(hunting): picked mon.noname-c con 0x7f76081430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7608135ac0 con 0x7f760813f710 2024-03-25T19:44:40.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7608137030 con 0x7f76080a98f0 2024-03-25T19:44:40.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76081372a0 con 0x7f76081430c0 2024-03-25T19:44:40.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:40.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.784+0000 7f761049a640 10 monclient(hunting): authenticate will time out at 5093.053027s 2024-03-25T19:44:40.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.792+0000 7f760e236640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f76080a98f0 0x7f7608133440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:54334/0 (socket says 172.21.15.69:54334) 2024-03-25T19:44:40.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.792+0000 7f760e236640 1 -- 172.21.15.69:0/2061924081 learned_addr learned my addr 172.21.15.69:0/2061924081 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:40.795 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.796+0000 7f75feffd640 10 client.?.objecter ms_handle_connect 0x7f76080a98f0 2024-03-25T19:44:40.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.796+0000 7f75feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:40.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.797+0000 7f75feffd640 10 client.?.objecter ms_handle_connect 0x7f760813f710 2024-03-25T19:44:40.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.797+0000 7f75feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:40.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.798+0000 7f75feffd640 10 client.?.objecter ms_handle_connect 0x7f76081430c0 2024-03-25T19:44:40.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.798+0000 7f75feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:40.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.812+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636711522 0 0) 0x7f7608135ac0 con 0x7f760813f710 2024-03-25T19:44:40.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.813+0000 7f75feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.813+0000 7f75feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.813+0000 7f75feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.813+0000 7f75feffd640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:44:40.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.813+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ec004eb0 con 0x7f760813f710 2024-03-25T19:44:40.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.873+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972216022 0 0) 0x7f76081372a0 con 0x7f76081430c0 2024-03-25T19:44:40.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.873+0000 7f75feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.873+0000 7f75feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.873+0000 7f75feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.873+0000 7f75feffd640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.873+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608135ac0 con 0x7f76081430c0 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.874+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3837577861 0 0) 0x7f75ec004eb0 con 0x7f760813f710 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.874+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76081372a0 con 0x7f760813f710 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.874+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f75f80018a0 con 0x7f760813f710 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.874+0000 7f75feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.874+0000 7f75feffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.874+0000 7f75feffd640 10 monclient(hunting): dump: 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:40.875 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:40.876 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:40.876 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.876 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:40.876 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:40.876 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:40.876 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:40.876 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:40.876 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.874+0000 7f75feffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:40.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.882+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2366096726 0 0) 0x7f7608135ac0 con 0x7f76081430c0 2024-03-25T19:44:40.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.882+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75ec004eb0 con 0x7f76081430c0 2024-03-25T19:44:40.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.884+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544664634 0 0) 0x7f7608137030 con 0x7f76080a98f0 2024-03-25T19:44:40.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.884+0000 7f75feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.884+0000 7f75feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.884+0000 7f75feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.884+0000 7f75feffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:44:40.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.884+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608135ac0 con 0x7f76080a98f0 2024-03-25T19:44:40.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.887+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7600001b60 con 0x7f76081430c0 2024-03-25T19:44:40.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.887+0000 7f75feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.887+0000 7f75feffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:40.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.887+0000 7f75feffd640 10 monclient(hunting): dump: 2024-03-25T19:44:40.887 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:40.887 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:40.887 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:40.887 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:40.888 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.888 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:40.888 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:40.888 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:40.888 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:40.888 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:40.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2346599805 0 0) 0x7f76081372a0 con 0x7f760813f710 2024-03-25T19:44:40.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 >> v1:172.21.15.69:6790/0 conn(0x7f76081430c0 legacy=0x7f76081e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 >> v1:172.21.15.169:6789/0 conn(0x7f76080a98f0 legacy=0x7f7608133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:40.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 1 monclient: found mon.a 2024-03-25T19:44:40.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7608137580 con 0x7f760813f710 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f75feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 5 monclient: authenticate success, global_id 4404 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 10 monclient: _renew_subs 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76081363b0 con 0x7f760813f710 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 10 monclient: _renew_subs 2024-03-25T19:44:40.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:40.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76081353a0 con 0x7f760813f710 2024-03-25T19:44:40.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.889+0000 7f761049a640 1 librados: init done 2024-03-25T19:44:40.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.890+0000 7f761049a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:40.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.890+0000 7f761049a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:40.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.890+0000 7f761049a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:40.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:40.890+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76080627b0 con 0x7f760813f710 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.216+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75f8001d10 con 0x7f760813f710 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.216+0000 7f75feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.216+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f75f8004a00 con 0x7f760813f710 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.216+0000 7f75feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.216+0000 7f75feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.216+0000 7f75feffd640 10 monclient: dump: 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:41.217 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:41.218 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:41.218 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:41.218 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:41.218 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:41.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.216+0000 7f760ca33640 4 set_mon_vals no callback set 2024-03-25T19:44:41.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.217+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f75f801aa90 con 0x7f760813f710 2024-03-25T19:44:41.219 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.218+0000 7f760e236640 10 monclient: get_auth_request con 0x7f75ec06df30 auth_method 0 2024-03-25T19:44:41.219 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.218+0000 7f75feffd640 10 client.4404.objecter ms_handle_connect 0x7f75ec06df30 2024-03-25T19:44:41.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.223+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f75f8054790 con 0x7f760813f710 2024-03-25T19:44:41.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.223+0000 7f75feffd640 10 client.4404.objecter ms_dispatch 0x7f7608000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:41.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.223+0000 7f75feffd640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:41.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.223+0000 7f75feffd640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:41.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.223+0000 7f75feffd640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T19:44:41.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.223+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75f80500d0 con 0x7f760813f710 2024-03-25T19:44:41.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.223+0000 7f75feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:41.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.223+0000 7f75feffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:41.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.465+0000 7f761049a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:41.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.465+0000 7f761049a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:41.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.465+0000 7f761049a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:44:41.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.466+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7608074cc0 con 0x7f760813f710 2024-03-25T19:44:41.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.524+0000 7f75feffd640 1 -- 172.21.15.69:0/2061924081 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f75f80521d0 con 0x7f760813f710 2024-03-25T19:44:41.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.524+0000 7f75feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:41.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.524+0000 7f75feffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:41.525 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:44:41.525 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:41.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 10 librados: watch_flush enter 2024-03-25T19:44:41.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 10 librados: watch_flush exit 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 10 client.4404.objecter successfully canceled tick 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 >> v1:172.21.15.169:6800/3072548259 conn(0x7f75ec06df30 legacy=0x7f75ec0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 10 monclient: shutdown 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 >> v1:172.21.15.69:6789/0 conn(0x7f760813f710 legacy=0x7f76081379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 shutdown_connections 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 >> 172.21.15.69:0/2061924081 conn(0x7f76080b7ea0 msgr2=0x7f7608074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 shutdown_connections 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 1 -- 172.21.15.69:0/2061924081 wait complete. 2024-03-25T19:44:41.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:41.527+0000 7f761049a640 1 librados: shutdown 2024-03-25T19:44:41.539 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:48.346 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:48.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.465+0000 7fe96ef3a640 10 monclient: get_monmap_and_config 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.465+0000 7fe96ef3a640 10 monclient: build_initial_monmap 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.465+0000 7fe96ef3a640 10 monclient: monmap: 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:44:48.466990+0000 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:44:48.466990+0000 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:44:48.466 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.466+0000 7fe96ef3a640 1 Processor -- start 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.466+0000 7fe96ef3a640 1 -- start start 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.466+0000 7fe96ef3a640 10 monclient: init 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 10 monclient(hunting): picked mon.noname-b con 0x7fe9680a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 10 monclient(hunting): picked mon.noname-c con 0x7fe96813f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 10 monclient(hunting): picked mon.noname-a con 0x7fe9681430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe968142520 con 0x7fe9681430c0 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680b28a0 con 0x7fe96813f710 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680b2a30 con 0x7fe9680a98f0 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.467+0000 7fe96ef3a640 10 monclient(hunting): authenticate will time out at 5100.736341s 2024-03-25T19:44:48.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.480+0000 7fe96d4d7640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe9681430c0 0x7fe96814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44280/0 (socket says 172.21.15.69:44280) 2024-03-25T19:44:48.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.480+0000 7fe96d4d7640 1 -- 172.21.15.69:0/1172251103 learned_addr learned my addr 172.21.15.69:0/1172251103 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.496+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161665490 0 0) 0x7fe968142520 con 0x7fe9681430c0 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.496+0000 7fe9677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.496+0000 7fe9677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.496+0000 7fe9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.496+0000 7fe9677fe640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:44:48.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.496+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe958003520 con 0x7fe9681430c0 2024-03-25T19:44:48.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.501+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046673482 0 0) 0x7fe9680b28a0 con 0x7fe96813f710 2024-03-25T19:44:48.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.501+0000 7fe9677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.501+0000 7fe9677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.501+0000 7fe9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.501+0000 7fe9677fe640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:44:48.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.501+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968142520 con 0x7fe96813f710 2024-03-25T19:44:48.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.507+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630663404 0 0) 0x7fe9680b2a30 con 0x7fe9680a98f0 2024-03-25T19:44:48.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.507+0000 7fe9677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.507+0000 7fe9677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.507+0000 7fe9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.507+0000 7fe9677fe640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:44:48.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.507+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9680b28a0 con 0x7fe9680a98f0 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459141966 0 0) 0x7fe958003520 con 0x7fe9681430c0 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 >> v1:172.21.15.69:6790/0 conn(0x7fe9680a98f0 legacy=0x7fe9680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 >> v1:172.21.15.169:6789/0 conn(0x7fe96813f710 legacy=0x7fe9681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 1 monclient: found mon.noname-a 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680b44c0 con 0x7fe9681430c0 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:48.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe960001a10 con 0x7fe9681430c0 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe9677fe640 10 monclient: dump: 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:48.522 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:48.523 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:48.523 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:48.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe96ef3a640 5 monclient: authenticate success, global_id 4407 2024-03-25T19:44:48.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.520+0000 7fe96ef3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:48.543 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.543+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe960001b70 con 0x7fe9681430c0 2024-03-25T19:44:48.543 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.543+0000 7fe9677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.543+0000 7fe9677fe640 1 -- 172.21.15.69:0/1172251103 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe960001f80 con 0x7fe9681430c0 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.543+0000 7fe96ef3a640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.543+0000 7fe96ef3a640 4 set_mon_vals no callback set 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.543+0000 7fe9677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.543+0000 7fe9677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.543+0000 7fe9677fe640 10 monclient: dump: 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:48.544 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:48.545 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:48.545 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.544+0000 7fe96ef3a640 10 monclient: shutdown 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.544+0000 7fe96ef3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.544+0000 7fe96ef3a640 1 -- 172.21.15.69:0/1172251103 >> v1:172.21.15.69:6789/0 conn(0x7fe9681430c0 legacy=0x7fe96814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.544+0000 7fe96ef3a640 1 -- 172.21.15.69:0/1172251103 shutdown_connections 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.544+0000 7fe96ef3a640 1 -- 172.21.15.69:0/1172251103 >> 172.21.15.69:0/1172251103 conn(0x7fe9680b7ea0 msgr2=0x7fe9680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.544+0000 7fe96ef3a640 1 -- 172.21.15.69:0/1172251103 shutdown_connections 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.544+0000 7fe96ef3a640 1 -- 172.21.15.69:0/1172251103 wait complete. 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 10 monclient: build_initial_monmap 2024-03-25T19:44:48.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 10 monclient: monmap: 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 1 librados: starting msgr at 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 1 librados: starting objecter 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 1 Processor -- start 2024-03-25T19:44:48.548 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 1 -- start start 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 1 librados: setting wanted keys 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 1 librados: calling monclient init 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.545+0000 7fe96ef3a640 10 monclient: init 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 10 monclient(hunting): picked mon.noname-c con 0x7fe9680a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 10 monclient(hunting): picked mon.noname-b con 0x7fe96813f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 10 monclient(hunting): picked mon.noname-a con 0x7fe9681430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe968136e20 con 0x7fe9681430c0 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680b7940 con 0x7fe96813f710 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9681351a0 con 0x7fe9680a98f0 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:48.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.546+0000 7fe96ef3a640 10 monclient(hunting): authenticate will time out at 5100.815316s 2024-03-25T19:44:48.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.557+0000 7fe967fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe96813f710 0x7fe9681db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:59140/0 (socket says 172.21.15.69:59140) 2024-03-25T19:44:48.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.557+0000 7fe967fff640 1 -- 172.21.15.69:0/3859722988 learned_addr learned my addr 172.21.15.69:0/3859722988 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:48.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.559+0000 7fe9657fa640 10 client.?.objecter ms_handle_connect 0x7fe9680a98f0 2024-03-25T19:44:48.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.559+0000 7fe9657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:48.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.565+0000 7fe9657fa640 10 client.?.objecter ms_handle_connect 0x7fe96813f710 2024-03-25T19:44:48.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.565+0000 7fe9657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:48.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.566+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180601476 0 0) 0x7fe9681351a0 con 0x7fe9680a98f0 2024-03-25T19:44:48.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.566+0000 7fe9657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.566+0000 7fe9657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.566+0000 7fe9657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.566+0000 7fe9657fa640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:44:48.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.566+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c004eb0 con 0x7fe9680a98f0 2024-03-25T19:44:48.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.566+0000 7fe9657fa640 10 client.?.objecter ms_handle_connect 0x7fe9681430c0 2024-03-25T19:44:48.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.566+0000 7fe9657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:48.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.578+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712716220 0 0) 0x7fe968136e20 con 0x7fe9681430c0 2024-03-25T19:44:48.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.578+0000 7fe9657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.578+0000 7fe9657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.578+0000 7fe9657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.578+0000 7fe9657fa640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:44:48.578 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.578+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9681351a0 con 0x7fe9681430c0 2024-03-25T19:44:48.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.579+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685142531 0 0) 0x7fe9680b7940 con 0x7fe96813f710 2024-03-25T19:44:48.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.579+0000 7fe9657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.579+0000 7fe9657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.579+0000 7fe9657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.579+0000 7fe9657fa640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:44:48.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.579+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968136e20 con 0x7fe96813f710 2024-03-25T19:44:48.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.582+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 739284768 0 0) 0x7fe94c004eb0 con 0x7fe9680a98f0 2024-03-25T19:44:48.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.582+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9680b7940 con 0x7fe9680a98f0 2024-03-25T19:44:48.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.582+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe95c0018f0 con 0x7fe9680a98f0 2024-03-25T19:44:48.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.582+0000 7fe9657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.582+0000 7fe9657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:48.582 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.582+0000 7fe9657fa640 10 monclient(hunting): dump: 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:48.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.582+0000 7fe9657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:48.587 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4258148846 0 0) 0x7fe9680b7940 con 0x7fe9680a98f0 2024-03-25T19:44:48.587 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 >> v1:172.21.15.169:6789/0 conn(0x7fe96813f710 legacy=0x7fe9681db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.587 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 >> v1:172.21.15.69:6789/0 conn(0x7fe9681430c0 legacy=0x7fe9681e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.587 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:48.587 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 1 monclient: found mon.c 2024-03-25T19:44:48.587 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968135330 con 0x7fe9680a98f0 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe9657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 10 monclient: _renew_subs 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe968135560 con 0x7fe9680a98f0 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 10 monclient: _renew_subs 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96813b490 con 0x7fe9680a98f0 2024-03-25T19:44:48.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.587+0000 7fe96ef3a640 1 librados: init done 2024-03-25T19:44:48.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.588+0000 7fe96ef3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:48.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.589+0000 7fe96ef3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:48.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.589+0000 7fe96ef3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:48.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.589+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9680b7ad0 con 0x7fe9680a98f0 2024-03-25T19:44:48.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.901+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe95c003610 con 0x7fe9680a98f0 2024-03-25T19:44:48.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.901+0000 7fe9657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:48.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.901+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe95c0039a0 con 0x7fe9680a98f0 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.901+0000 7fe9657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.901+0000 7fe9657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.901+0000 7fe9657fa640 10 monclient: dump: 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:48.902 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:48.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.901+0000 7fe966ffd640 4 set_mon_vals no callback set 2024-03-25T19:44:48.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.902+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fe95c003bc0 con 0x7fe9680a98f0 2024-03-25T19:44:48.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.902+0000 7fe967fff640 10 monclient: get_auth_request con 0x7fe94c06ddf0 auth_method 0 2024-03-25T19:44:48.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.903+0000 7fe9657fa640 10 client.4394.objecter ms_handle_connect 0x7fe94c06ddf0 2024-03-25T19:44:48.907 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.907+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fe95c053640 con 0x7fe9680a98f0 2024-03-25T19:44:48.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.907+0000 7fe9657fa640 10 client.4394.objecter ms_dispatch 0x7fe968000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:48.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.907+0000 7fe9657fa640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:48.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.907+0000 7fe9657fa640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:48.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.907+0000 7fe9657fa640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T19:44:48.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.907+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe95c04f150 con 0x7fe9680a98f0 2024-03-25T19:44:48.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.907+0000 7fe9657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:48.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:48.907+0000 7fe9657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:49.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.127+0000 7fe96ef3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:49.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.127+0000 7fe96ef3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:49.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.127+0000 7fe96ef3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:44:49.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.127+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9680b7730 con 0x7fe9680a98f0 2024-03-25T19:44:49.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.153+0000 7fe9657fa640 1 -- 172.21.15.69:0/3859722988 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fe95c051080 con 0x7fe9680a98f0 2024-03-25T19:44:49.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.153+0000 7fe9657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:49.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.153+0000 7fe9657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:49.153 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:44:49.153 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:49.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.155+0000 7fe96ef3a640 10 librados: watch_flush enter 2024-03-25T19:44:49.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.155+0000 7fe96ef3a640 10 librados: watch_flush exit 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.155+0000 7fe96ef3a640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.155+0000 7fe96ef3a640 10 client.4394.objecter successfully canceled tick 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.155+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 >> v1:172.21.15.169:6800/3072548259 conn(0x7fe94c06ddf0 legacy=0x7fe94c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.155+0000 7fe96ef3a640 10 monclient: shutdown 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.155+0000 7fe96ef3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.155+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 >> v1:172.21.15.69:6790/0 conn(0x7fe9680a98f0 legacy=0x7fe9681daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.156+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 shutdown_connections 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.156+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 >> 172.21.15.69:0/3859722988 conn(0x7fe9680b7ea0 msgr2=0x7fe968074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.156+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 shutdown_connections 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.156+0000 7fe96ef3a640 1 -- 172.21.15.69:0/3859722988 wait complete. 2024-03-25T19:44:49.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:49.156+0000 7fe96ef3a640 1 librados: shutdown 2024-03-25T19:44:49.168 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:55.975 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.090+0000 7f4e0a817640 10 monclient: get_monmap_and_config 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.090+0000 7f4e0a817640 10 monclient: build_initial_monmap 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.090+0000 7f4e0a817640 10 monclient: monmap: 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:44:56.090499+0000 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:44:56.090499+0000 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:56.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 1 Processor -- start 2024-03-25T19:44:56.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 1 -- start start 2024-03-25T19:44:56.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 10 monclient: init 2024-03-25T19:44:56.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 10 monclient(hunting): picked mon.noname-b con 0x7f4e040a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 10 monclient(hunting): picked mon.noname-a con 0x7f4e041430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e04142520 con 0x7f4e041430c0 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.091+0000 7f4e0a817640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040b28a0 con 0x7f4e0413f710 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.092+0000 7f4e0a817640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040b2a30 con 0x7f4e040a98f0 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.092+0000 7f4e0a817640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:56.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.092+0000 7f4e0a817640 10 monclient(hunting): authenticate will time out at 5108.359627s 2024-03-25T19:44:56.102 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.103+0000 7f4e08db4640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4e041430c0 0x7f4e0414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:34252/0 (socket says 172.21.15.69:34252) 2024-03-25T19:44:56.102 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.103+0000 7f4e08db4640 1 -- 172.21.15.69:0/2071419269 learned_addr learned my addr 172.21.15.69:0/2071419269 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:56.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.114+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410339928 0 0) 0x7f4e040b28a0 con 0x7f4e0413f710 2024-03-25T19:44:56.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.114+0000 7f4e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.114+0000 7f4e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.114+0000 7f4e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.114+0000 7f4e02ffd640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:44:56.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.114+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dec003520 con 0x7f4e0413f710 2024-03-25T19:44:56.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2176115971 0 0) 0x7f4dec003520 con 0x7f4e0413f710 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 >> v1:172.21.15.69:6790/0 conn(0x7f4e040a98f0 legacy=0x7f4e040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 >> v1:172.21.15.69:6789/0 conn(0x7f4e041430c0 legacy=0x7f4e0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 1 monclient: found mon.noname-c 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e040b44c0 con 0x7f4e0413f710 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4df8001690 con 0x7f4e0413f710 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e02ffd640 10 monclient: dump: 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e0a817640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:44:56.127 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.126+0000 7f4e0a817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:56.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.157+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4df80017f0 con 0x7f4e0413f710 2024-03-25T19:44:56.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.157+0000 7f4e02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:56.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.157+0000 7f4e02ffd640 1 -- 172.21.15.69:0/2071419269 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4df8001c00 con 0x7f4e0413f710 2024-03-25T19:44:56.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.157+0000 7f4e0a817640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:56.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.157+0000 7f4e0a817640 4 set_mon_vals no callback set 2024-03-25T19:44:56.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.157+0000 7f4e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.157+0000 7f4e02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.157+0000 7f4e02ffd640 10 monclient: dump: 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:56.157 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.158+0000 7f4e0a817640 10 monclient: shutdown 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.158+0000 7f4e0a817640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.158+0000 7f4e0a817640 1 -- 172.21.15.69:0/2071419269 >> v1:172.21.15.169:6789/0 conn(0x7f4e0413f710 legacy=0x7f4e041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.158+0000 7f4e0a817640 1 -- 172.21.15.69:0/2071419269 shutdown_connections 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.158+0000 7f4e0a817640 1 -- 172.21.15.69:0/2071419269 >> 172.21.15.69:0/2071419269 conn(0x7f4e040b7ea0 msgr2=0x7f4e040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.158+0000 7f4e0a817640 1 -- 172.21.15.69:0/2071419269 shutdown_connections 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.158+0000 7f4e0a817640 1 -- 172.21.15.69:0/2071419269 wait complete. 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 10 monclient: build_initial_monmap 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 10 monclient: monmap: 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:56.159 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 1 librados: starting msgr at 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 1 librados: starting objecter 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 1 Processor -- start 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 1 -- start start 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 1 librados: setting wanted keys 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 1 librados: calling monclient init 2024-03-25T19:44:56.160 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.159+0000 7f4e0a817640 10 monclient: init 2024-03-25T19:44:56.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:56.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:56.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 10 monclient(hunting): picked mon.noname-c con 0x7f4e040a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:44:56.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:44:56.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 10 monclient(hunting): picked mon.noname-a con 0x7f4e041430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:44:56.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e041370a0 con 0x7f4e041430c0 2024-03-25T19:44:56.161 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e04135090 con 0x7f4e0413f710 2024-03-25T19:44:56.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e04135220 con 0x7f4e040a98f0 2024-03-25T19:44:56.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:56.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.160+0000 7f4e0a817640 10 monclient(hunting): authenticate will time out at 5108.428453s 2024-03-25T19:44:56.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.164+0000 7f4e08db4640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4e041430c0 0x7f4e041d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:34332/0 (socket says 172.21.15.69:34332) 2024-03-25T19:44:56.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.164+0000 7f4e08db4640 1 -- 172.21.15.69:0/1760217985 learned_addr learned my addr 172.21.15.69:0/1760217985 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:44:56.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.169+0000 7f4e00ff9640 10 client.?.objecter ms_handle_connect 0x7f4e040a98f0 2024-03-25T19:44:56.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.169+0000 7f4e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:56.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.170+0000 7f4e00ff9640 10 client.?.objecter ms_handle_connect 0x7f4e041430c0 2024-03-25T19:44:56.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.170+0000 7f4e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:56.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.180+0000 7f4e00ff9640 10 client.?.objecter ms_handle_connect 0x7f4e0413f710 2024-03-25T19:44:56.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.180+0000 7f4e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:56.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.180+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869926500 0 0) 0x7f4e041370a0 con 0x7f4e041430c0 2024-03-25T19:44:56.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.180+0000 7f4e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.180+0000 7f4e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.180+0000 7f4e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.180+0000 7f4e00ff9640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:44:56.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.180+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df0004eb0 con 0x7f4e041430c0 2024-03-25T19:44:56.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.183+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979184893 0 0) 0x7f4e04135220 con 0x7f4e040a98f0 2024-03-25T19:44:56.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.183+0000 7f4e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.183+0000 7f4e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.183+0000 7f4e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.183+0000 7f4e00ff9640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:44:56.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.183+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e041370a0 con 0x7f4e040a98f0 2024-03-25T19:44:56.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.196+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287391302 0 0) 0x7f4e04135090 con 0x7f4e0413f710 2024-03-25T19:44:56.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.196+0000 7f4e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.196+0000 7f4e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.196+0000 7f4e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.196+0000 7f4e00ff9640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:44:56.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.196+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e04135220 con 0x7f4e0413f710 2024-03-25T19:44:56.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.206+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546723708 0 0) 0x7f4df0004eb0 con 0x7f4e041430c0 2024-03-25T19:44:56.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.206+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e04135090 con 0x7f4e041430c0 2024-03-25T19:44:56.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.206+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4dfc001b10 con 0x7f4e041430c0 2024-03-25T19:44:56.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.206+0000 7f4e00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.206+0000 7f4e00ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:56.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.206+0000 7f4e00ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:56.205 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:56.205 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:56.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.206+0000 7f4e00ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:56.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.213+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1965868576 0 0) 0x7f4e041370a0 con 0x7f4e040a98f0 2024-03-25T19:44:56.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.213+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4df0004eb0 con 0x7f4e040a98f0 2024-03-25T19:44:56.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.213+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4df40018b0 con 0x7f4e040a98f0 2024-03-25T19:44:56.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.213+0000 7f4e00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.213+0000 7f4e00ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:56.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.213+0000 7f4e00ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:56.213 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:56.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.216+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1965411814 0 0) 0x7f4e04135220 con 0x7f4e0413f710 2024-03-25T19:44:56.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.216+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e041370a0 con 0x7f4e0413f710 2024-03-25T19:44:56.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.216+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4df8001b00 con 0x7f4e0413f710 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.216+0000 7f4e00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.216+0000 7f4e00ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.216+0000 7f4e00ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:56.216 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1568660008 0 0) 0x7f4e041370a0 con 0x7f4e0413f710 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 >> v1:172.21.15.69:6790/0 conn(0x7f4e040a98f0 legacy=0x7f4e041dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 >> v1:172.21.15.69:6789/0 conn(0x7f4e041430c0 legacy=0x7f4e041d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 1 monclient: found mon.b 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e041d5000 con 0x7f4e0413f710 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:56.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 5 monclient: authenticate success, global_id 4393 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 10 monclient: _renew_subs 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e041d52b0 con 0x7f4e0413f710 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 10 monclient: _renew_subs 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e04135910 con 0x7f4e0413f710 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.234+0000 7f4e0a817640 1 librados: init done 2024-03-25T19:44:56.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.235+0000 7f4e0a817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:56.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.235+0000 7f4e0a817640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:56.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.235+0000 7f4e0a817640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:56.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.235+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e040627b0 con 0x7f4e0413f710 2024-03-25T19:44:56.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.560+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4df80038a0 con 0x7f4e0413f710 2024-03-25T19:44:56.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.560+0000 7f4e00ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:56.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.560+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4df8003bb0 con 0x7f4e0413f710 2024-03-25T19:44:56.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.560+0000 7f4e00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.559 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.560+0000 7f4e00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.560+0000 7f4e00ff9640 10 monclient: dump: 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:44:56.560 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.560+0000 7f4e027fc640 4 set_mon_vals no callback set 2024-03-25T19:44:56.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.561+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f4df8019b80 con 0x7f4e0413f710 2024-03-25T19:44:56.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.561+0000 7f4e03fff640 10 monclient: get_auth_request con 0x7f4df006e110 auth_method 0 2024-03-25T19:44:56.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.562+0000 7f4e00ff9640 10 client.4393.objecter ms_handle_connect 0x7f4df006e110 2024-03-25T19:44:56.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.566+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f4df8053650 con 0x7f4e0413f710 2024-03-25T19:44:56.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.566+0000 7f4e00ff9640 10 client.4393.objecter ms_dispatch 0x7f4e04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:56.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.566+0000 7f4e00ff9640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:56.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.566+0000 7f4e00ff9640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:56.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.566+0000 7f4e00ff9640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T19:44:56.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.567+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4df804f160 con 0x7f4e0413f710 2024-03-25T19:44:56.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.567+0000 7f4e00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:56.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.567+0000 7f4e00ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:56.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.791+0000 7f4e0a817640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:56.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.791+0000 7f4e0a817640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:56.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.791+0000 7f4e0a817640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:44:56.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.791+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e040008d0 con 0x7f4e0413f710 2024-03-25T19:44:56.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.892+0000 7f4e00ff9640 1 -- 172.21.15.69:0/1760217985 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f4df8051090 con 0x7f4e0413f710 2024-03-25T19:44:56.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.892+0000 7f4e00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:56.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.892+0000 7f4e00ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:56.892 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:44:56.892 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 10 librados: watch_flush enter 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 10 librados: watch_flush exit 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 10 client.4393.objecter successfully canceled tick 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 >> v1:172.21.15.169:6800/3072548259 conn(0x7f4df006e110 legacy=0x7f4df00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 10 monclient: shutdown 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 >> v1:172.21.15.169:6789/0 conn(0x7f4e0413f710 legacy=0x7f4e041d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 shutdown_connections 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 >> 172.21.15.69:0/1760217985 conn(0x7f4e040b7ea0 msgr2=0x7f4e04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 shutdown_connections 2024-03-25T19:44:56.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 1 -- 172.21.15.69:0/1760217985 wait complete. 2024-03-25T19:44:56.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:44:56.895+0000 7f4e0a817640 1 librados: shutdown 2024-03-25T19:44:56.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:02.019 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:39.983 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.073 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:40.037 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.106 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:40.070 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.140 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:40.094 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.146 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:40.100 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.151 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:40.115 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.155 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:40.119 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.179 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:40.144 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.180 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:40.134 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:03.712 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.829+0000 7fcbbc664640 10 monclient: get_monmap_and_config 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.829+0000 7fcbbc664640 10 monclient: build_initial_monmap 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.829+0000 7fcbbc664640 10 monclient: monmap: 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:45:03.831027+0000 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:45:03.831027+0000 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:03.830 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:45:03.831 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:45:03.831 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:03.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.830+0000 7fcbbc664640 1 Processor -- start 2024-03-25T19:45:03.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.831+0000 7fcbbc664640 1 -- start start 2024-03-25T19:45:03.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.831+0000 7fcbbc664640 10 monclient: init 2024-03-25T19:45:03.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.831+0000 7fcbbc664640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:03.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:03.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 10 monclient(hunting): picked mon.noname-c con 0x7fcbb40a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:03.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 10 monclient(hunting): picked mon.noname-b con 0x7fcbb413f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:03.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 10 monclient(hunting): picked mon.noname-a con 0x7fcbb41430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:03.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb4142520 con 0x7fcbb41430c0 2024-03-25T19:45:03.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb40b28a0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.832 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb40b2a30 con 0x7fcbb413f710 2024-03-25T19:45:03.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:03.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.832+0000 7fcbbc664640 10 monclient(hunting): authenticate will time out at 5116.100876s 2024-03-25T19:45:03.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.845+0000 7fcbbac01640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcbb41430c0 0x7fcbb414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33382/0 (socket says 172.21.15.69:33382) 2024-03-25T19:45:03.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.845+0000 7fcbbac01640 1 -- 172.21.15.69:0/1266270248 learned_addr learned my addr 172.21.15.69:0/1266270248 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:03.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.859+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061603493 0 0) 0x7fcbb40b2a30 con 0x7fcbb413f710 2024-03-25T19:45:03.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.859+0000 7fcbb93fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.859+0000 7fcbb93fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.859+0000 7fcbb93fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.859+0000 7fcbb93fe640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:45:03.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.859+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba0003520 con 0x7fcbb413f710 2024-03-25T19:45:03.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.862+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213741376 0 0) 0x7fcbb4142520 con 0x7fcbb41430c0 2024-03-25T19:45:03.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.862+0000 7fcbb93fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.862+0000 7fcbb93fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.862+0000 7fcbb93fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.862+0000 7fcbb93fe640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:45:03.862 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.862+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb40b2a30 con 0x7fcbb41430c0 2024-03-25T19:45:03.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.873+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093475297 0 0) 0x7fcbb40b28a0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.873+0000 7fcbb93fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.873+0000 7fcbb93fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.873+0000 7fcbb93fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.873+0000 7fcbb93fe640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:45:03.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.873+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb4142520 con 0x7fcbb40a98f0 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.890+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 868960025 0 0) 0x7fcbb4142520 con 0x7fcbb40a98f0 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.890+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 >> v1:172.21.15.69:6790/0 conn(0x7fcbb413f710 legacy=0x7fcbb41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.890+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 >> v1:172.21.15.69:6789/0 conn(0x7fcbb41430c0 legacy=0x7fcbb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.890+0000 7fcbb93fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.890+0000 7fcbb93fe640 1 monclient: found mon.noname-c 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40b44c0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.891 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcbb00015b0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbb93fe640 10 monclient: dump: 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:03.892 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:03.893 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:03.893 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:03.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbbc664640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:45:03.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.891+0000 7fcbbc664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:03.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.899+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbb0001710 con 0x7fcbb40a98f0 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.899+0000 7fcbb93fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.899+0000 7fcbb93fe640 1 -- 172.21.15.69:0/1266270248 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcbb0001b60 con 0x7fcbb40a98f0 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.899+0000 7fcbbc664640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.899+0000 7fcbbc664640 4 set_mon_vals no callback set 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.899+0000 7fcbb93fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.899+0000 7fcbb93fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.899+0000 7fcbb93fe640 10 monclient: dump: 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:03.900 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:03.901 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:03.901 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:03.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.900+0000 7fcbbc664640 10 monclient: shutdown 2024-03-25T19:45:03.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.900+0000 7fcbbc664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:03.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.900+0000 7fcbbc664640 1 -- 172.21.15.69:0/1266270248 >> v1:172.21.15.169:6789/0 conn(0x7fcbb40a98f0 legacy=0x7fcbb40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.900+0000 7fcbbc664640 1 -- 172.21.15.69:0/1266270248 shutdown_connections 2024-03-25T19:45:03.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.900+0000 7fcbbc664640 1 -- 172.21.15.69:0/1266270248 >> 172.21.15.69:0/1266270248 conn(0x7fcbb40b7ea0 msgr2=0x7fcbb40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:03.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.900+0000 7fcbbc664640 1 -- 172.21.15.69:0/1266270248 shutdown_connections 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.900+0000 7fcbbc664640 1 -- 172.21.15.69:0/1266270248 wait complete. 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.901+0000 7fcbbc664640 10 monclient: build_initial_monmap 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.901+0000 7fcbbc664640 10 monclient: monmap: 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:03.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.902+0000 7fcbbc664640 1 librados: starting msgr at 2024-03-25T19:45:03.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.902+0000 7fcbbc664640 1 librados: starting objecter 2024-03-25T19:45:03.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.902+0000 7fcbbc664640 1 Processor -- start 2024-03-25T19:45:03.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.902+0000 7fcbbc664640 1 -- start start 2024-03-25T19:45:03.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.902+0000 7fcbbc664640 1 librados: setting wanted keys 2024-03-25T19:45:03.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.902+0000 7fcbbc664640 1 librados: calling monclient init 2024-03-25T19:45:03.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.902+0000 7fcbbc664640 10 monclient: init 2024-03-25T19:45:03.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:03.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:03.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 10 monclient(hunting): picked mon.noname-a con 0x7fcbb40a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:03.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 10 monclient(hunting): picked mon.noname-c con 0x7fcbb413f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:03.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 10 monclient(hunting): picked mon.noname-b con 0x7fcbb41430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:03.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb41391a0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb40abda0 con 0x7fcbb41430c0 2024-03-25T19:45:03.906 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb4139460 con 0x7fcbb413f710 2024-03-25T19:45:03.906 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:03.906 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.903+0000 7fcbbc664640 10 monclient(hunting): authenticate will time out at 5116.172017s 2024-03-25T19:45:03.907 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.907+0000 7fcbba400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcbb40a98f0 0x7fcbb41367f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33460/0 (socket says 172.21.15.69:33460) 2024-03-25T19:45:03.907 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.907+0000 7fcbba400640 1 -- 172.21.15.69:0/889844398 learned_addr learned my addr 172.21.15.69:0/889844398 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:03.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.910+0000 7fcbaaffd640 10 client.?.objecter ms_handle_connect 0x7fcbb40a98f0 2024-03-25T19:45:03.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.910+0000 7fcbaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.913+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525607655 0 0) 0x7fcbb41391a0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.913+0000 7fcbaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.913+0000 7fcbaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.913+0000 7fcbaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.913+0000 7fcbaaffd640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:45:03.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.913+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb98004eb0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.925+0000 7fcbaaffd640 10 client.?.objecter ms_handle_connect 0x7fcbb41430c0 2024-03-25T19:45:03.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.925+0000 7fcbaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.926+0000 7fcbaaffd640 10 client.?.objecter ms_handle_connect 0x7fcbb413f710 2024-03-25T19:45:03.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.926+0000 7fcbaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.929+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1454533798 0 0) 0x7fcb98004eb0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.929+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbb41391a0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.929+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcbb0001ae0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.929+0000 7fcbaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.929+0000 7fcbaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.929+0000 7fcbaaffd640 10 monclient(hunting): dump: 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:03.930 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:03.931 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:03.931 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.929+0000 7fcbaaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.942+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60763226 0 0) 0x7fcbb40abda0 con 0x7fcbb41430c0 2024-03-25T19:45:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.942+0000 7fcbaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.942+0000 7fcbaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.942+0000 7fcbaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.942+0000 7fcbaaffd640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:45:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.942+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb98004eb0 con 0x7fcbb41430c0 2024-03-25T19:45:03.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.944+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172647997 0 0) 0x7fcbb4139460 con 0x7fcbb413f710 2024-03-25T19:45:03.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.944+0000 7fcbaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.944+0000 7fcbaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.944+0000 7fcbaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.944+0000 7fcbaaffd640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:45:03.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.944+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb40abda0 con 0x7fcbb413f710 2024-03-25T19:45:03.945 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2976190915 0 0) 0x7fcbb41391a0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 >> v1:172.21.15.69:6790/0 conn(0x7fcbb413f710 legacy=0x7fcbb40ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 >> v1:172.21.15.169:6789/0 conn(0x7fcbb41430c0 legacy=0x7fcbb4138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 1 monclient: found mon.a 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40abff0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 5 monclient: authenticate success, global_id 4422 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 10 monclient: _renew_subs 2024-03-25T19:45:03.946 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:03.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb40ac2d0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:03.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 10 monclient: _renew_subs 2024-03-25T19:45:03.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:03.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb41e34b0 con 0x7fcbb40a98f0 2024-03-25T19:45:03.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.945+0000 7fcbbc664640 1 librados: init done 2024-03-25T19:45:03.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.946+0000 7fcbbc664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:03.948 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.946+0000 7fcbbc664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:03.948 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.946+0000 7fcbbc664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:03.948 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:03.946+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb40627b0 con 0x7fcbb40a98f0 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.253+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbb00038a0 con 0x7fcbb40a98f0 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.253+0000 7fcbaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.253+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fcbb0003cf0 con 0x7fcbb40a98f0 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.253+0000 7fcbaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.253+0000 7fcbaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.253+0000 7fcbaaffd640 10 monclient: dump: 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:04.254 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:04.255 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:04.255 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:04.255 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:04.255 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:04.255 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:04.255 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:04.255 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:04.255 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.253+0000 7fcbb8bfd640 4 set_mon_vals no callback set 2024-03-25T19:45:04.256 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.254+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fcbb0003f10 con 0x7fcbb40a98f0 2024-03-25T19:45:04.256 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.255+0000 7fcbb9bff640 10 monclient: get_auth_request con 0x7fcb9806de10 auth_method 0 2024-03-25T19:45:04.256 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.255+0000 7fcbaaffd640 10 client.4422.objecter ms_handle_connect 0x7fcb9806de10 2024-03-25T19:45:04.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.262+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fcbb0054900 con 0x7fcbb40a98f0 2024-03-25T19:45:04.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.262+0000 7fcbaaffd640 10 client.4422.objecter ms_dispatch 0x7fcbb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:04.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.262+0000 7fcbaaffd640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:04.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.262+0000 7fcbaaffd640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:04.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.262+0000 7fcbaaffd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T19:45:04.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.262+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbb0050410 con 0x7fcbb40a98f0 2024-03-25T19:45:04.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.262+0000 7fcbaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:04.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.262+0000 7fcbaaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:04.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.500+0000 7fcbbc664640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:04.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.500+0000 7fcbbc664640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:04.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.500+0000 7fcbbc664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:04.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.500+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbb413db40 con 0x7fcbb40a98f0 2024-03-25T19:45:04.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.562+0000 7fcbaaffd640 1 -- 172.21.15.69:0/889844398 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fcbb0052340 con 0x7fcbb40a98f0 2024-03-25T19:45:04.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.562+0000 7fcbaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:04.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.562+0000 7fcbaaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:04.562 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:45:04.562 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 10 librados: watch_flush enter 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 10 librados: watch_flush exit 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 10 client.4422.objecter successfully canceled tick 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 >> v1:172.21.15.169:6800/3072548259 conn(0x7fcb9806de10 legacy=0x7fcb980702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 10 monclient: shutdown 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 >> v1:172.21.15.69:6789/0 conn(0x7fcbb40a98f0 legacy=0x7fcbb41367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:04.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 shutdown_connections 2024-03-25T19:45:04.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 >> 172.21.15.69:0/889844398 conn(0x7fcbb40b7ea0 msgr2=0x7fcbb4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:04.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 shutdown_connections 2024-03-25T19:45:04.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.564+0000 7fcbbc664640 1 -- 172.21.15.69:0/889844398 wait complete. 2024-03-25T19:45:04.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:04.565+0000 7fcbbc664640 1 librados: shutdown 2024-03-25T19:45:04.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:11.384 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:11.494 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.494+0000 7f4495aff640 10 monclient: get_monmap_and_config 2024-03-25T19:45:11.494 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.494+0000 7f4495aff640 10 monclient: build_initial_monmap 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.494+0000 7f4495aff640 10 monclient: monmap: 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:45:11.495827+0000 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:45:11.495827+0000 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:11.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 1 Processor -- start 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 1 -- start start 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 10 monclient: init 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 10 monclient(hunting): picked mon.noname-c con 0x7f44900b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 10 monclient(hunting): picked mon.noname-b con 0x7f44900b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 10 monclient(hunting): picked mon.noname-a con 0x7f449014be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.495+0000 7f4495aff640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44900af940 con 0x7f449014be90 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.496+0000 7f4495aff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44900b0a70 con 0x7f44900b4d60 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.496+0000 7f4495aff640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44900b0d50 con 0x7f44900b1730 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.496+0000 7f4495aff640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:11.497 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.496+0000 7f4495aff640 10 monclient(hunting): authenticate will time out at 5123.764638s 2024-03-25T19:45:11.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.530+0000 7f448f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f44900b4d60 0x7f449013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:49226/0 (socket says 172.21.15.69:49226) 2024-03-25T19:45:11.531 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.531+0000 7f448f7fe640 1 -- 172.21.15.69:0/2944235276 learned_addr learned my addr 172.21.15.69:0/2944235276 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:11.551 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.552+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661992543 0 0) 0x7f44900b0a70 con 0x7f44900b4d60 2024-03-25T19:45:11.552 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.552+0000 7f448e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.552 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.552+0000 7f448e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.552 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.552+0000 7f448e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.552 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.552+0000 7f448e7fc640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:45:11.552 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.552+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c003520 con 0x7f44900b4d60 2024-03-25T19:45:11.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.555+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625233446 0 0) 0x7f44900b0d50 con 0x7f44900b1730 2024-03-25T19:45:11.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.555+0000 7f448e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.555+0000 7f448e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.555+0000 7f448e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.555+0000 7f448e7fc640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:45:11.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.555+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44900b0a70 con 0x7f44900b1730 2024-03-25T19:45:11.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.562+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151162646 0 0) 0x7f44900af940 con 0x7f449014be90 2024-03-25T19:45:11.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.562+0000 7f448e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.562+0000 7f448e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.562+0000 7f448e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.562+0000 7f448e7fc640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:45:11.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.562+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44900b0d50 con 0x7f449014be90 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4033202422 0 0) 0x7f447c003520 con 0x7f44900b4d60 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 >> v1:172.21.15.69:6790/0 conn(0x7f44900b1730 legacy=0x7f44900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 >> v1:172.21.15.69:6789/0 conn(0x7f449014be90 legacy=0x7f449014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44900b1030 con 0x7f44900b4d60 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4480001a10 con 0x7f44900b4d60 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f4495aff640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f4495aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f448e7fc640 10 monclient: dump: 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:11.566 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.564+0000 7f4495aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:11.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.575+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4480001b70 con 0x7f44900b4d60 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.575+0000 7f448e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.575+0000 7f448e7fc640 1 -- 172.21.15.69:0/2944235276 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4480001f80 con 0x7f44900b4d60 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.575+0000 7f4495aff640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.575+0000 7f4495aff640 4 set_mon_vals no callback set 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.575+0000 7f448e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.575+0000 7f448e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.575+0000 7f448e7fc640 10 monclient: dump: 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.576 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:11.577 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:11.577 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:11.577 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:11.577 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.576+0000 7f4495aff640 10 monclient: shutdown 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.576+0000 7f4495aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.576+0000 7f4495aff640 1 -- 172.21.15.69:0/2944235276 >> v1:172.21.15.169:6789/0 conn(0x7f44900b4d60 legacy=0x7f449013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.576+0000 7f4495aff640 1 -- 172.21.15.69:0/2944235276 shutdown_connections 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.576+0000 7f4495aff640 1 -- 172.21.15.69:0/2944235276 >> 172.21.15.69:0/2944235276 conn(0x7f44900617a0 msgr2=0x7f4490061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.576+0000 7f4495aff640 1 -- 172.21.15.69:0/2944235276 shutdown_connections 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.576+0000 7f4495aff640 1 -- 172.21.15.69:0/2944235276 wait complete. 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 10 monclient: build_initial_monmap 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 10 monclient: monmap: 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:11.579 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 1 librados: starting msgr at 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 1 librados: starting objecter 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 1 Processor -- start 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 1 -- start start 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 1 librados: setting wanted keys 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 1 librados: calling monclient init 2024-03-25T19:45:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.577+0000 7f4495aff640 10 monclient: init 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 10 monclient(hunting): picked mon.noname-c con 0x7f44900b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 10 monclient(hunting): picked mon.noname-b con 0x7f44900b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 10 monclient(hunting): picked mon.noname-a con 0x7f449014be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44901440c0 con 0x7f449014be90 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449013f020 con 0x7f44900b4d60 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4490136d30 con 0x7f44900b1730 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:11.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.578+0000 7f4495aff640 10 monclient(hunting): authenticate will time out at 5123.847157s 2024-03-25T19:45:11.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.590+0000 7f448f7fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f44900b1730 0x7f44901439b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:45494/0 (socket says 172.21.15.69:45494) 2024-03-25T19:45:11.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.590+0000 7f448f7fe640 1 -- 172.21.15.69:0/3830916184 learned_addr learned my addr 172.21.15.69:0/3830916184 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:11.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.591+0000 7f4487fff640 10 client.?.objecter ms_handle_connect 0x7f44900b1730 2024-03-25T19:45:11.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.591+0000 7f4487fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:11.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.597+0000 7f4487fff640 10 client.?.objecter ms_handle_connect 0x7f449014be90 2024-03-25T19:45:11.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.597+0000 7f4487fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:11.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.598+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638358958 0 0) 0x7f4490136d30 con 0x7f44900b1730 2024-03-25T19:45:11.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.598+0000 7f4487fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.598+0000 7f4487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.598+0000 7f4487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.598+0000 7f4487fff640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:45:11.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.598+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4470004eb0 con 0x7f44900b1730 2024-03-25T19:45:11.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.600+0000 7f4487fff640 10 client.?.objecter ms_handle_connect 0x7f44900b4d60 2024-03-25T19:45:11.600 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.600+0000 7f4487fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:11.601 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.601+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314007904 0 0) 0x7f44901440c0 con 0x7f449014be90 2024-03-25T19:45:11.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.601+0000 7f4487fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.601+0000 7f4487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.601+0000 7f4487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.601+0000 7f4487fff640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:45:11.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.601+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4490136d30 con 0x7f449014be90 2024-03-25T19:45:11.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.607+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 57702455 0 0) 0x7f4470004eb0 con 0x7f44900b1730 2024-03-25T19:45:11.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.607+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44901440c0 con 0x7f44900b1730 2024-03-25T19:45:11.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.607+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532583090 0 0) 0x7f449013f020 con 0x7f44900b4d60 2024-03-25T19:45:11.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.607+0000 7f4487fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.607+0000 7f4487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.607+0000 7f4487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.607+0000 7f4487fff640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:45:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.607+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4470004eb0 con 0x7f44900b4d60 2024-03-25T19:45:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.608+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4480001fe0 con 0x7f44900b1730 2024-03-25T19:45:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.608+0000 7f4487fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.608+0000 7f4487fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.608+0000 7f4487fff640 10 monclient(hunting): dump: 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:11.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.608+0000 7f4487fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:11.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.610+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2925919368 0 0) 0x7f4490136d30 con 0x7f449014be90 2024-03-25T19:45:11.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.610+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f449013f020 con 0x7f449014be90 2024-03-25T19:45:11.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.616+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3681454803 0 0) 0x7f44901440c0 con 0x7f44900b1730 2024-03-25T19:45:11.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.616+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 >> v1:172.21.15.169:6789/0 conn(0x7f44900b4d60 legacy=0x7f44901d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 >> v1:172.21.15.69:6789/0 conn(0x7f449014be90 legacy=0x7f44901d8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:11.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 1 monclient: found mon.c 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490136ec0 con 0x7f44900b1730 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 5 monclient: authenticate success, global_id 4412 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 10 monclient: _renew_subs 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44901e33d0 con 0x7f44900b1730 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:11.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 10 monclient: _renew_subs 2024-03-25T19:45:11.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:11.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44901e3740 con 0x7f44900b1730 2024-03-25T19:45:11.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.617+0000 7f4495aff640 1 librados: init done 2024-03-25T19:45:11.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.618+0000 7f4495aff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:11.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.618+0000 7f4495aff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:11.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.618+0000 7f4495aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:11.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.618+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44901d79a0 con 0x7f44900b1730 2024-03-25T19:45:11.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.920+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4480004e10 con 0x7f44900b1730 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.920+0000 7f4487fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.920+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4480005120 con 0x7f44900b1730 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.920+0000 7f4487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.920+0000 7f4487fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.920+0000 7f4487fff640 10 monclient: dump: 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:11.921 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:11.922 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:11.922 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:11.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.920+0000 7f448dffb640 4 set_mon_vals no callback set 2024-03-25T19:45:11.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.921+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f448001b0f0 con 0x7f44900b1730 2024-03-25T19:45:11.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.922+0000 7f448effd640 10 monclient: get_auth_request con 0x7f447006e080 auth_method 0 2024-03-25T19:45:11.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.922+0000 7f4487fff640 10 client.4412.objecter ms_handle_connect 0x7f447006e080 2024-03-25T19:45:11.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.928+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f4480054de0 con 0x7f44900b1730 2024-03-25T19:45:11.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.928+0000 7f4487fff640 10 client.4412.objecter ms_dispatch 0x7f4490000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:11.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.928+0000 7f4487fff640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:11.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.928+0000 7f4487fff640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:11.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.928+0000 7f4487fff640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T19:45:11.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.928+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4480050720 con 0x7f44900b1730 2024-03-25T19:45:11.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.928+0000 7f4487fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:11.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:11.928+0000 7f4487fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:12.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.171+0000 7f4495aff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:12.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.171+0000 7f4495aff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:12.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.171+0000 7f4495aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:12.171 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.171+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44901401f0 con 0x7f44900b1730 2024-03-25T19:45:12.198 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.198+0000 7f4487fff640 1 -- 172.21.15.69:0/3830916184 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f4480052820 con 0x7f44900b1730 2024-03-25T19:45:12.198 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.198+0000 7f4487fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:12.198 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.198+0000 7f4487fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:12.198 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:45:12.198 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:12.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 10 librados: watch_flush enter 2024-03-25T19:45:12.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 10 librados: watch_flush exit 2024-03-25T19:45:12.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T19:45:12.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 10 client.4412.objecter successfully canceled tick 2024-03-25T19:45:12.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 >> v1:172.21.15.169:6800/3072548259 conn(0x7f447006e080 legacy=0x7f4470070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:12.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 10 monclient: shutdown 2024-03-25T19:45:12.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:12.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 >> v1:172.21.15.69:6790/0 conn(0x7f44900b1730 legacy=0x7f44901439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:12.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.200+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 shutdown_connections 2024-03-25T19:45:12.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.201+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 >> 172.21.15.69:0/3830916184 conn(0x7f44900617a0 msgr2=0x7f449013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:12.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.201+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 shutdown_connections 2024-03-25T19:45:12.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.201+0000 7f4495aff640 1 -- 172.21.15.69:0/3830916184 wait complete. 2024-03-25T19:45:12.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:12.201+0000 7f4495aff640 1 librados: shutdown 2024-03-25T19:45:12.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:19.019 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:19.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.135+0000 7fbe9ce57640 10 monclient: get_monmap_and_config 2024-03-25T19:45:19.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.135+0000 7fbe9ce57640 10 monclient: build_initial_monmap 2024-03-25T19:45:19.136 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.135+0000 7fbe9ce57640 10 monclient: monmap: 2024-03-25T19:45:19.136 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:19.136 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:19.136 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:45:19.136535+0000 2024-03-25T19:45:19.136 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:45:19.136535+0000 2024-03-25T19:45:19.136 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:19.136 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:19.137 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:19.137 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:45:19.137 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:45:19.137 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:19.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.135+0000 7fbe9ce57640 1 Processor -- start 2024-03-25T19:45:19.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 1 -- start start 2024-03-25T19:45:19.137 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 10 monclient: init 2024-03-25T19:45:19.138 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:19.138 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:19.138 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 10 monclient(hunting): picked mon.noname-b con 0x7fbe980a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 10 monclient(hunting): picked mon.noname-a con 0x7fbe9813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 10 monclient(hunting): picked mon.noname-c con 0x7fbe981430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe98142520 con 0x7fbe9813f710 2024-03-25T19:45:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe980b28a0 con 0x7fbe981430c0 2024-03-25T19:45:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe980b2a30 con 0x7fbe980a98f0 2024-03-25T19:45:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.136+0000 7fbe9ce57640 10 monclient(hunting): authenticate will time out at 5131.405584s 2024-03-25T19:45:19.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.145+0000 7fbe96d9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fbe981430c0 0x7fbe9814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:53980/0 (socket says 172.21.15.69:53980) 2024-03-25T19:45:19.145 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.145+0000 7fbe96d9d640 1 -- 172.21.15.69:0/4090426698 learned_addr learned my addr 172.21.15.69:0/4090426698 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:19.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.156+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512965144 0 0) 0x7fbe980b2a30 con 0x7fbe980a98f0 2024-03-25T19:45:19.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.156+0000 7fbe9559a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.156+0000 7fbe9559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.156+0000 7fbe9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.156+0000 7fbe9559a640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:45:19.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.156+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe80003520 con 0x7fbe980a98f0 2024-03-25T19:45:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.158+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647650585 0 0) 0x7fbe980b28a0 con 0x7fbe981430c0 2024-03-25T19:45:19.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.158+0000 7fbe9559a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.158+0000 7fbe9559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.158+0000 7fbe9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.158+0000 7fbe9559a640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:45:19.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.158+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe980b2a30 con 0x7fbe981430c0 2024-03-25T19:45:19.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2050499578 0 0) 0x7fbe80003520 con 0x7fbe980a98f0 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 >> v1:172.21.15.169:6789/0 conn(0x7fbe981430c0 legacy=0x7fbe9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 >> v1:172.21.15.69:6789/0 conn(0x7fbe9813f710 legacy=0x7fbe981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 10 monclient: _finish_hunting 0 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 1 monclient: found mon.noname-b 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980b44c0 con 0x7fbe980a98f0 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 10 monclient: _finish_auth 0 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 10 monclient: _check_auth_tickets 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:19.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fbe88001690 con 0x7fbe980a98f0 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9559a640 10 monclient: dump: 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:19.169 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:19.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9ce57640 5 monclient: authenticate success, global_id 4415 2024-03-25T19:45:19.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.166+0000 7fbe9ce57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:19.175 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.175+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe880017f0 con 0x7fbe980a98f0 2024-03-25T19:45:19.175 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.175+0000 7fbe9559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.175+0000 7fbe9559a640 1 -- 172.21.15.69:0/4090426698 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fbe88001c00 con 0x7fbe980a98f0 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.175+0000 7fbe9ce57640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.175+0000 7fbe9ce57640 4 set_mon_vals no callback set 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.175+0000 7fbe9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.175+0000 7fbe9559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.175+0000 7fbe9559a640 10 monclient: dump: 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:19.176 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:19.177 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:19.177 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.176+0000 7fbe9ce57640 10 monclient: shutdown 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.176+0000 7fbe9ce57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.176+0000 7fbe9ce57640 1 -- 172.21.15.69:0/4090426698 >> v1:172.21.15.69:6790/0 conn(0x7fbe980a98f0 legacy=0x7fbe980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.176+0000 7fbe9ce57640 1 -- 172.21.15.69:0/4090426698 shutdown_connections 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.176+0000 7fbe9ce57640 1 -- 172.21.15.69:0/4090426698 >> 172.21.15.69:0/4090426698 conn(0x7fbe980b7ea0 msgr2=0x7fbe980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.176+0000 7fbe9ce57640 1 -- 172.21.15.69:0/4090426698 shutdown_connections 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.176+0000 7fbe9ce57640 1 -- 172.21.15.69:0/4090426698 wait complete. 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.177+0000 7fbe9ce57640 10 monclient: build_initial_monmap 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.177+0000 7fbe9ce57640 10 monclient: monmap: 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:19.179 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 librados: starting msgr at 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 librados: starting objecter 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 Processor -- start 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 -- start start 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 librados: setting wanted keys 2024-03-25T19:45:19.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 librados: calling monclient init 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 10 monclient: init 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 10 monclient(hunting): picked mon.noname-b con 0x7fbe980a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 10 monclient(hunting): picked mon.noname-a con 0x7fbe9813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 10 monclient(hunting): picked mon.noname-c con 0x7fbe981430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe98138b10 con 0x7fbe9813f710 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe98136e10 con 0x7fbe980a98f0 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe98138d60 con 0x7fbe981430c0 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.178+0000 7fbe9ce57640 10 monclient(hunting): authenticate will time out at 5131.447559s 2024-03-25T19:45:19.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.190+0000 7fbe96d9d640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fbe981430c0 0x7fbe981e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:50302/0 (socket says 172.21.15.69:50302) 2024-03-25T19:45:19.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.190+0000 7fbe96d9d640 1 -- 172.21.15.69:0/2126421158 learned_addr learned my addr 172.21.15.69:0/2126421158 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:19.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.192+0000 7fbe86ffd640 10 client.?.objecter ms_handle_connect 0x7fbe981430c0 2024-03-25T19:45:19.192 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.192+0000 7fbe86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.197+0000 7fbe86ffd640 10 client.?.objecter ms_handle_connect 0x7fbe9813f710 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.197+0000 7fbe86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:19.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.201+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545921814 0 0) 0x7fbe98138d60 con 0x7fbe981430c0 2024-03-25T19:45:19.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.201+0000 7fbe86ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.201+0000 7fbe86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.201+0000 7fbe86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.201+0000 7fbe86ffd640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:45:19.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.201+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe78004eb0 con 0x7fbe981430c0 2024-03-25T19:45:19.208 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.208+0000 7fbe86ffd640 10 client.?.objecter ms_handle_connect 0x7fbe980a98f0 2024-03-25T19:45:19.208 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.208+0000 7fbe86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.214+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 112394050 0 0) 0x7fbe78004eb0 con 0x7fbe981430c0 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.215+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe98138d60 con 0x7fbe981430c0 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.215+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fbe90001d10 con 0x7fbe981430c0 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.215+0000 7fbe86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.215+0000 7fbe86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.215+0000 7fbe86ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:19.216 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:19.216 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.216 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:19.216 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:19.216 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:19.216 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:19.216 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:19.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.215+0000 7fbe86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:19.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.220+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528503232 0 0) 0x7fbe98138b10 con 0x7fbe9813f710 2024-03-25T19:45:19.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.220+0000 7fbe86ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.220+0000 7fbe86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.220+0000 7fbe86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.220+0000 7fbe86ffd640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:45:19.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.220+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe78004eb0 con 0x7fbe9813f710 2024-03-25T19:45:19.223 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.223+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486967218 0 0) 0x7fbe98136e10 con 0x7fbe980a98f0 2024-03-25T19:45:19.224 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.223+0000 7fbe86ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.224 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.223+0000 7fbe86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.224 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.223+0000 7fbe86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.224 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.223+0000 7fbe86ffd640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:45:19.224 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.223+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe98138b10 con 0x7fbe980a98f0 2024-03-25T19:45:19.225 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 976675219 0 0) 0x7fbe98138d60 con 0x7fbe981430c0 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 >> v1:172.21.15.169:6789/0 conn(0x7fbe980a98f0 legacy=0x7fbe98137560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 >> v1:172.21.15.69:6789/0 conn(0x7fbe9813f710 legacy=0x7fbe981df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 1 monclient: found mon.c 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe98139040 con 0x7fbe981430c0 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 5 monclient: authenticate success, global_id 4418 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 10 monclient: _renew_subs 2024-03-25T19:45:19.226 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:19.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe98139380 con 0x7fbe981430c0 2024-03-25T19:45:19.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:19.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 10 monclient: _renew_subs 2024-03-25T19:45:19.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:19.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe98139920 con 0x7fbe981430c0 2024-03-25T19:45:19.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.225+0000 7fbe9ce57640 1 librados: init done 2024-03-25T19:45:19.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.226+0000 7fbe9ce57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:19.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.226+0000 7fbe9ce57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:19.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.226+0000 7fbe9ce57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:19.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.226+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe980627b0 con 0x7fbe981430c0 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.519+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe90004b60 con 0x7fbe981430c0 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.519+0000 7fbe86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.519+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fbe90004e70 con 0x7fbe981430c0 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.519+0000 7fbe86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.519+0000 7fbe86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.519+0000 7fbe86ffd640 10 monclient: dump: 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:19.520 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:19.521 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:19.521 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:19.521 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:19.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.519+0000 7fbe94d99640 4 set_mon_vals no callback set 2024-03-25T19:45:19.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.520+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fbe9001ae40 con 0x7fbe981430c0 2024-03-25T19:45:19.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.521+0000 7fbe9659c640 10 monclient: get_auth_request con 0x7fbe7806de00 auth_method 0 2024-03-25T19:45:19.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.521+0000 7fbe86ffd640 10 client.4418.objecter ms_handle_connect 0x7fbe7806de00 2024-03-25T19:45:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.526+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fbe90054c50 con 0x7fbe981430c0 2024-03-25T19:45:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.526+0000 7fbe86ffd640 10 client.4418.objecter ms_dispatch 0x7fbe98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:19.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.526+0000 7fbe86ffd640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:19.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.526+0000 7fbe86ffd640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:19.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.526+0000 7fbe86ffd640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-25T19:45:19.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.526+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe90050590 con 0x7fbe981430c0 2024-03-25T19:45:19.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.526+0000 7fbe86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:19.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.526+0000 7fbe86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:19.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.781+0000 7fbe9ce57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:19.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.781+0000 7fbe9ce57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:19.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.781+0000 7fbe9ce57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:19.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.781+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe981391d0 con 0x7fbe981430c0 2024-03-25T19:45:19.807 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.807+0000 7fbe86ffd640 1 -- 172.21.15.69:0/2126421158 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fbe90052690 con 0x7fbe981430c0 2024-03-25T19:45:19.807 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.807+0000 7fbe86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:19.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.807+0000 7fbe86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:19.808 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:45:19.808 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:19.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 10 librados: watch_flush enter 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 10 librados: watch_flush exit 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 10 client.4418.objecter successfully canceled tick 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 >> v1:172.21.15.169:6800/3072548259 conn(0x7fbe7806de00 legacy=0x7fbe780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 10 monclient: shutdown 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 >> v1:172.21.15.69:6790/0 conn(0x7fbe981430c0 legacy=0x7fbe981e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 shutdown_connections 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 >> 172.21.15.69:0/2126421158 conn(0x7fbe980b7ea0 msgr2=0x7fbe98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 shutdown_connections 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 1 -- 172.21.15.69:0/2126421158 wait complete. 2024-03-25T19:45:19.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:19.810+0000 7fbe9ce57640 1 librados: shutdown 2024-03-25T19:45:19.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:22.142 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:00.106 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.149 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:00.113 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.190 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:00.154 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.205 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:00.169 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.212 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:00.177 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.213 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:00.168 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.216 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:00.171 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.249 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:00.213 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.250 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:00.204 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:26.627 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:26.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.739+0000 7fae8ae38640 10 monclient: get_monmap_and_config 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.739+0000 7fae8ae38640 10 monclient: build_initial_monmap 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.739+0000 7fae8ae38640 10 monclient: monmap: 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:45:26.740960+0000 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:45:26.740960+0000 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:45:26.740 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:26.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.740+0000 7fae8ae38640 1 Processor -- start 2024-03-25T19:45:26.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.740+0000 7fae8ae38640 1 -- start start 2024-03-25T19:45:26.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.740+0000 7fae8ae38640 10 monclient: init 2024-03-25T19:45:26.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:26.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:26.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 10 monclient(hunting): picked mon.noname-a con 0x7fae840a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:26.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 10 monclient(hunting): picked mon.noname-b con 0x7fae8413f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:26.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 10 monclient(hunting): picked mon.noname-c con 0x7fae841430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:26.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae84142520 con 0x7fae840a98f0 2024-03-25T19:45:26.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae840b28a0 con 0x7fae841430c0 2024-03-25T19:45:26.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae840b2a30 con 0x7fae8413f710 2024-03-25T19:45:26.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:26.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.741+0000 7fae8ae38640 10 monclient(hunting): authenticate will time out at 5139.009913s 2024-03-25T19:45:26.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.743+0000 7fae88bd4640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fae840a98f0 0x7fae840a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35896/0 (socket says 172.21.15.69:35896) 2024-03-25T19:45:26.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.743+0000 7fae88bd4640 1 -- 172.21.15.69:0/4122885020 learned_addr learned my addr 172.21.15.69:0/4122885020 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:26.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.754+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449344944 0 0) 0x7fae84142520 con 0x7fae840a98f0 2024-03-25T19:45:26.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.754+0000 7fae837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.754+0000 7fae837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.754+0000 7fae837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.754+0000 7fae837fe640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:45:26.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.754+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae74003520 con 0x7fae840a98f0 2024-03-25T19:45:26.760 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.760+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722693365 0 0) 0x7fae840b28a0 con 0x7fae841430c0 2024-03-25T19:45:26.760 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.760+0000 7fae837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.760 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.760+0000 7fae837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.760+0000 7fae837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.760+0000 7fae837fe640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:45:26.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.760+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae84142520 con 0x7fae841430c0 2024-03-25T19:45:26.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.761+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088475670 0 0) 0x7fae840b2a30 con 0x7fae8413f710 2024-03-25T19:45:26.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.761+0000 7fae837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.762 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.761+0000 7fae837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.762 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.761+0000 7fae837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.762 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.761+0000 7fae837fe640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:45:26.762 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.761+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae840b28a0 con 0x7fae8413f710 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1745933171 0 0) 0x7fae74003520 con 0x7fae840a98f0 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 >> v1:172.21.15.69:6790/0 conn(0x7fae8413f710 legacy=0x7fae841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 >> v1:172.21.15.169:6789/0 conn(0x7fae841430c0 legacy=0x7fae8414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 1 monclient: found mon.noname-a 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae840b44c0 con 0x7fae840a98f0 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:26.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fae78001690 con 0x7fae840a98f0 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae837fe640 10 monclient: dump: 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae8ae38640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:45:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.766+0000 7fae8ae38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:26.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.781+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae780017f0 con 0x7fae840a98f0 2024-03-25T19:45:26.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:26.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae837fe640 1 -- 172.21.15.69:0/4122885020 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fae78001c00 con 0x7fae840a98f0 2024-03-25T19:45:26.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:26.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 4 set_mon_vals no callback set 2024-03-25T19:45:26.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:26.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae837fe640 10 monclient: dump: 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:26.783 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:26.784 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:26.784 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 10 monclient: shutdown 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 1 -- 172.21.15.69:0/4122885020 >> v1:172.21.15.69:6789/0 conn(0x7fae840a98f0 legacy=0x7fae840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 1 -- 172.21.15.69:0/4122885020 shutdown_connections 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 1 -- 172.21.15.69:0/4122885020 >> 172.21.15.69:0/4122885020 conn(0x7fae840b7ea0 msgr2=0x7fae840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 1 -- 172.21.15.69:0/4122885020 shutdown_connections 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.782+0000 7fae8ae38640 1 -- 172.21.15.69:0/4122885020 wait complete. 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.783+0000 7fae8ae38640 10 monclient: build_initial_monmap 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.783+0000 7fae8ae38640 10 monclient: monmap: 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:26.787 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:45:26.788 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:45:26.788 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:26.788 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:45:26.788 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:26.788 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:45:26.788 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.784+0000 7fae8ae38640 1 librados: starting msgr at 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.784+0000 7fae8ae38640 1 librados: starting objecter 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.784+0000 7fae8ae38640 1 Processor -- start 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.784+0000 7fae8ae38640 1 -- start start 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.784+0000 7fae8ae38640 1 librados: setting wanted keys 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.784+0000 7fae8ae38640 1 librados: calling monclient init 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.784+0000 7fae8ae38640 10 monclient: init 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:26.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:26.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 10 monclient(hunting): picked mon.noname-a con 0x7fae840a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:26.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 10 monclient(hunting): picked mon.noname-b con 0x7fae8413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:26.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 10 monclient(hunting): picked mon.noname-c con 0x7fae841430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:26.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae840abd30 con 0x7fae840a98f0 2024-03-25T19:45:26.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae841d77a0 con 0x7fae8413f710 2024-03-25T19:45:26.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae841d51f0 con 0x7fae841430c0 2024-03-25T19:45:26.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:26.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.785+0000 7fae8ae38640 10 monclient(hunting): authenticate will time out at 5139.054129s 2024-03-25T19:45:26.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.792+0000 7fae83fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fae8413f710 0x7fae84138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:33894/0 (socket says 172.21.15.69:33894) 2024-03-25T19:45:26.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.792+0000 7fae83fff640 1 -- 172.21.15.69:0/1285306474 learned_addr learned my addr 172.21.15.69:0/1285306474 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:26.795 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.795+0000 7fae817fa640 10 client.?.objecter ms_handle_connect 0x7fae841430c0 2024-03-25T19:45:26.795 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.795+0000 7fae817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:26.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.797+0000 7fae817fa640 10 client.?.objecter ms_handle_connect 0x7fae8413f710 2024-03-25T19:45:26.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.797+0000 7fae817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:26.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.801+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358104190 0 0) 0x7fae841d51f0 con 0x7fae841430c0 2024-03-25T19:45:26.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.801+0000 7fae817fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.801+0000 7fae817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.801+0000 7fae817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.802 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.801+0000 7fae817fa640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:45:26.802 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.801+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae68004eb0 con 0x7fae841430c0 2024-03-25T19:45:26.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.810+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609533322 0 0) 0x7fae841d77a0 con 0x7fae8413f710 2024-03-25T19:45:26.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.810+0000 7fae817fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.810+0000 7fae817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.810+0000 7fae817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.810+0000 7fae817fa640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:45:26.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.810+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae841d51f0 con 0x7fae8413f710 2024-03-25T19:45:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.822+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2593555666 0 0) 0x7fae68004eb0 con 0x7fae841430c0 2024-03-25T19:45:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.822+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae841d77a0 con 0x7fae841430c0 2024-03-25T19:45:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.822+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fae7c001bd0 con 0x7fae841430c0 2024-03-25T19:45:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.822+0000 7fae817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.822+0000 7fae817fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.822+0000 7fae817fa640 10 monclient(hunting): dump: 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:26.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.822+0000 7fae817fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:26.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.826+0000 7fae817fa640 10 client.?.objecter ms_handle_connect 0x7fae840a98f0 2024-03-25T19:45:26.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.826+0000 7fae817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:26.830 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.830+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1439788996 0 0) 0x7fae841d51f0 con 0x7fae8413f710 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.830+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae68004eb0 con 0x7fae8413f710 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.830+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fae700019f0 con 0x7fae8413f710 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.830+0000 7fae817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.830+0000 7fae817fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.830+0000 7fae817fa640 10 monclient(hunting): dump: 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:26.831 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1851278289 0 0) 0x7fae841d77a0 con 0x7fae841430c0 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 >> v1:172.21.15.169:6789/0 conn(0x7fae8413f710 legacy=0x7fae84138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 >> v1:172.21.15.69:6789/0 conn(0x7fae840a98f0 legacy=0x7fae841d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 1 monclient: found mon.c 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae841d83d0 con 0x7fae841430c0 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 5 monclient: authenticate success, global_id 4424 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 10 monclient: _renew_subs 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae84138a50 con 0x7fae841430c0 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 10 monclient: _renew_subs 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae841e35c0 con 0x7fae841430c0 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.837+0000 7fae8ae38640 1 librados: init done 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.838+0000 7fae8ae38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.838+0000 7fae8ae38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.838+0000 7fae8ae38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:26.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:26.838+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae840627b0 con 0x7fae841430c0 2024-03-25T19:45:27.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.139+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae7c003970 con 0x7fae841430c0 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.139+0000 7fae817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.139+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fae7c002100 con 0x7fae841430c0 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.139+0000 7fae817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.139+0000 7fae817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.139+0000 7fae817fa640 10 monclient: dump: 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:27.140 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:27.141 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:27.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.139+0000 7fae82ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:27.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.140+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fae7c019bb0 con 0x7fae841430c0 2024-03-25T19:45:27.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.141+0000 7fae88bd4640 10 monclient: get_auth_request con 0x7fae6806de40 auth_method 0 2024-03-25T19:45:27.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.141+0000 7fae817fa640 10 client.4424.objecter ms_handle_connect 0x7fae6806de40 2024-03-25T19:45:27.148 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.148+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fae7c0538e0 con 0x7fae841430c0 2024-03-25T19:45:27.148 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.148+0000 7fae817fa640 10 client.4424.objecter ms_dispatch 0x7fae84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:27.148 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.148+0000 7fae817fa640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:27.148 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.148+0000 7fae817fa640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:27.148 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.148+0000 7fae817fa640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T19:45:27.148 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.148+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae7c04f220 con 0x7fae841430c0 2024-03-25T19:45:27.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.148+0000 7fae817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:27.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.148+0000 7fae817fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:27.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.400+0000 7fae8ae38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:27.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.400+0000 7fae8ae38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:27.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.400+0000 7fae8ae38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:45:27.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.400+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae84074cc0 con 0x7fae841430c0 2024-03-25T19:45:27.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.420+0000 7fae817fa640 1 -- 172.21.15.69:0/1285306474 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fae7c051320 con 0x7fae841430c0 2024-03-25T19:45:27.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.420+0000 7fae817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:27.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.420+0000 7fae817fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:27.420 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:45:27.420 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.422+0000 7fae8ae38640 10 librados: watch_flush enter 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.422+0000 7fae8ae38640 10 librados: watch_flush exit 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.422+0000 7fae8ae38640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.422+0000 7fae8ae38640 10 client.4424.objecter successfully canceled tick 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.422+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 >> v1:172.21.15.169:6800/3072548259 conn(0x7fae6806de40 legacy=0x7fae68070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.422+0000 7fae8ae38640 10 monclient: shutdown 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.422+0000 7fae8ae38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.422+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 >> v1:172.21.15.69:6790/0 conn(0x7fae841430c0 legacy=0x7fae841e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.423+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 shutdown_connections 2024-03-25T19:45:27.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.423+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 >> 172.21.15.69:0/1285306474 conn(0x7fae840b7ea0 msgr2=0x7fae84074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:27.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.423+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 shutdown_connections 2024-03-25T19:45:27.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.423+0000 7fae8ae38640 1 -- 172.21.15.69:0/1285306474 wait complete. 2024-03-25T19:45:27.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:27.423+0000 7fae8ae38640 1 librados: shutdown 2024-03-25T19:45:27.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:34.242 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:34.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.358+0000 7fb2ac139640 10 monclient: get_monmap_and_config 2024-03-25T19:45:34.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.358+0000 7fb2ac139640 10 monclient: build_initial_monmap 2024-03-25T19:45:34.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.358+0000 7fb2ac139640 10 monclient: monmap: 2024-03-25T19:45:34.357 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:34.357 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:34.358 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:45:34.358536+0000 2024-03-25T19:45:34.358 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:45:34.358536+0000 2024-03-25T19:45:34.358 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:34.358 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:34.358 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:34.358 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:45:34.358 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:45:34.358 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:34.359 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.359+0000 7fb2ac139640 1 Processor -- start 2024-03-25T19:45:34.359 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.359+0000 7fb2ac139640 1 -- start start 2024-03-25T19:45:34.359 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.359+0000 7fb2ac139640 10 monclient: init 2024-03-25T19:45:34.359 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:34.359 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:34.359 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a40a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:34.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a413f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:34.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a41430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:34.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a4142520 con 0x7fb2a41430c0 2024-03-25T19:45:34.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a40b28a0 con 0x7fb2a40a98f0 2024-03-25T19:45:34.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a40b2a30 con 0x7fb2a413f710 2024-03-25T19:45:34.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:34.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.360+0000 7fb2ac139640 10 monclient(hunting): authenticate will time out at 5146.628218s 2024-03-25T19:45:34.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.372+0000 7fb2a96d4640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fb2a413f710 0x7fb2a41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:34086/0 (socket says 172.21.15.69:34086) 2024-03-25T19:45:34.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.372+0000 7fb2a96d4640 1 -- 172.21.15.69:0/2622936788 learned_addr learned my addr 172.21.15.69:0/2622936788 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:34.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.385+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351454488 0 0) 0x7fb2a40b2a30 con 0x7fb2a413f710 2024-03-25T19:45:34.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.385+0000 7fb2a8ed3640 10 monclient(hunting): _negotiate 2024-03-25T19:45:34.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.385+0000 7fb2a8ed3640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:34.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.385+0000 7fb2a8ed3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:34.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.385+0000 7fb2a8ed3640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:45:34.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.385+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb290003520 con 0x7fb2a413f710 2024-03-25T19:45:34.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.391+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821506567 0 0) 0x7fb2a4142520 con 0x7fb2a41430c0 2024-03-25T19:45:34.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.391+0000 7fb2a8ed3640 10 monclient(hunting): _negotiate 2024-03-25T19:45:34.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.391+0000 7fb2a8ed3640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:34.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.391+0000 7fb2a8ed3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:34.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.391+0000 7fb2a8ed3640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:45:34.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.391+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a40b2a30 con 0x7fb2a41430c0 2024-03-25T19:45:34.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.396+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503127038 0 0) 0x7fb2a40b28a0 con 0x7fb2a40a98f0 2024-03-25T19:45:34.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.396+0000 7fb2a8ed3640 10 monclient(hunting): _negotiate 2024-03-25T19:45:34.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.396+0000 7fb2a8ed3640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:34.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.396+0000 7fb2a8ed3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:34.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.396+0000 7fb2a8ed3640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:45:34.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.396+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a4142520 con 0x7fb2a40a98f0 2024-03-25T19:45:34.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044147617 0 0) 0x7fb290003520 con 0x7fb2a413f710 2024-03-25T19:45:34.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 >> v1:172.21.15.169:6789/0 conn(0x7fb2a40a98f0 legacy=0x7fb2a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 >> v1:172.21.15.69:6789/0 conn(0x7fb2a41430c0 legacy=0x7fb2a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 10 monclient: _finish_hunting 0 2024-03-25T19:45:34.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 1 monclient: found mon.noname-b 2024-03-25T19:45:34.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a40b44c0 con 0x7fb2a413f710 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 10 monclient: _finish_auth 0 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 10 monclient: _check_auth_tickets 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb294001690 con 0x7fb2a413f710 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2ac139640 5 monclient: authenticate success, global_id 4427 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2ac139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2a8ed3640 10 monclient: dump: 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:34.405 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:34.406 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:34.406 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:34.406 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:34.406 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:34.406 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:34.406 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:34.406 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:34.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.405+0000 7fb2ac139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2940017f0 con 0x7fb2a413f710 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2a8ed3640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2a8ed3640 1 -- 172.21.15.69:0/2622936788 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb294001c00 con 0x7fb2a413f710 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2ac139640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2ac139640 4 set_mon_vals no callback set 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2a8ed3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2a8ed3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2a8ed3640 10 monclient: dump: 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:34.412 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:34.413 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:34.413 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:34.413 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:34.413 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:34.413 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:34.413 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:34.413 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.413+0000 7fb2ac139640 10 monclient: shutdown 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.414+0000 7fb2ac139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.414+0000 7fb2ac139640 1 -- 172.21.15.69:0/2622936788 >> v1:172.21.15.69:6790/0 conn(0x7fb2a413f710 legacy=0x7fb2a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.414+0000 7fb2ac139640 1 -- 172.21.15.69:0/2622936788 shutdown_connections 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.414+0000 7fb2ac139640 1 -- 172.21.15.69:0/2622936788 >> 172.21.15.69:0/2622936788 conn(0x7fb2a40b7ea0 msgr2=0x7fb2a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.414+0000 7fb2ac139640 1 -- 172.21.15.69:0/2622936788 shutdown_connections 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.414+0000 7fb2ac139640 1 -- 172.21.15.69:0/2622936788 wait complete. 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.414+0000 7fb2ac139640 10 monclient: build_initial_monmap 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.414+0000 7fb2ac139640 10 monclient: monmap: 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:34.415 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 librados: starting msgr at 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 librados: starting objecter 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 Processor -- start 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 -- start start 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 librados: setting wanted keys 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 librados: calling monclient init 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 10 monclient: init 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:34.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:34.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a40a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:34.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:34.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a41430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:34.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a41370a0 con 0x7fb2a40a98f0 2024-03-25T19:45:34.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a4135090 con 0x7fb2a413f710 2024-03-25T19:45:34.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a4135220 con 0x7fb2a41430c0 2024-03-25T19:45:34.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:34.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.415+0000 7fb2ac139640 10 monclient(hunting): authenticate will time out at 5146.683429s 2024-03-25T19:45:34.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.424+0000 7fb2aa6d6640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fb2a41430c0 0x7fb2a41d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:34102/0 (socket says 172.21.15.69:34102) 2024-03-25T19:45:34.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.424+0000 7fb2aa6d6640 1 -- 172.21.15.69:0/824578096 learned_addr learned my addr 172.21.15.69:0/824578096 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:34.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.424+0000 7fb29a7fc640 10 client.?.objecter ms_handle_connect 0x7fb2a41430c0 2024-03-25T19:45:34.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.424+0000 7fb29a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:34.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.427+0000 7fb29a7fc640 10 client.?.objecter ms_handle_connect 0x7fb2a413f710 2024-03-25T19:45:34.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.427+0000 7fb29a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:34.432 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.433+0000 7fb29a7fc640 10 client.?.objecter ms_handle_connect 0x7fb2a40a98f0 2024-03-25T19:45:34.432 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.433+0000 7fb29a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:34.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.435+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527296733 0 0) 0x7fb2a4135090 con 0x7fb2a413f710 2024-03-25T19:45:34.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.435+0000 7fb29a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:34.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.435+0000 7fb29a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:34.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.435+0000 7fb29a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:34.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.435+0000 7fb29a7fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:45:34.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.435+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb288004eb0 con 0x7fb2a413f710 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.436+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507887037 0 0) 0x7fb2a4135220 con 0x7fb2a41430c0 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.436+0000 7fb29a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.436+0000 7fb29a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.436+0000 7fb29a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.436+0000 7fb29a7fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.436+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a4135090 con 0x7fb2a41430c0 2024-03-25T19:45:34.442 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.444+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146412165 0 0) 0x7fb2a41370a0 con 0x7fb2a40a98f0 2024-03-25T19:45:34.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.444+0000 7fb29a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:34.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.444+0000 7fb29a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:34.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.444+0000 7fb29a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:34.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.444+0000 7fb29a7fc640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:45:34.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.444+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a4135220 con 0x7fb2a40a98f0 2024-03-25T19:45:34.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.452+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387107197 0 0) 0x7fb288004eb0 con 0x7fb2a413f710 2024-03-25T19:45:34.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.452+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2a41370a0 con 0x7fb2a413f710 2024-03-25T19:45:34.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.452+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb294001b00 con 0x7fb2a413f710 2024-03-25T19:45:34.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.452+0000 7fb29a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:34.451 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.452+0000 7fb29a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.452+0000 7fb29a7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:34.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.452+0000 7fb29a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.458+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2771189240 0 0) 0x7fb2a41370a0 con 0x7fb2a413f710 2024-03-25T19:45:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.458+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 >> v1:172.21.15.69:6790/0 conn(0x7fb2a41430c0 legacy=0x7fb2a41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.458+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 >> v1:172.21.15.69:6789/0 conn(0x7fb2a40a98f0 legacy=0x7fb2a41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb29a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb29a7fc640 1 monclient: found mon.b 2024-03-25T19:45:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb29a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb29a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:45:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a41d5000 con 0x7fb2a413f710 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb29a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb29a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb29a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 5 monclient: authenticate success, global_id 4423 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 10 monclient: _renew_subs 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a41d52b0 con 0x7fb2a413f710 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 10 monclient: _renew_subs 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a4135910 con 0x7fb2a413f710 2024-03-25T19:45:34.459 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.459+0000 7fb2ac139640 1 librados: init done 2024-03-25T19:45:34.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.460+0000 7fb2ac139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:34.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.460+0000 7fb2ac139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:34.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.460+0000 7fb2ac139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:45:34.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.460+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a40627b0 con 0x7fb2a413f710 2024-03-25T19:45:34.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.679+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb294001220 con 0x7fb2a413f710 2024-03-25T19:45:34.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.679+0000 7fb29a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:34.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.679+0000 7fb29bfff640 4 set_mon_vals no callback set 2024-03-25T19:45:34.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.738+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb294004c30 con 0x7fb2a413f710 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.738+0000 7fb29a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.738+0000 7fb29a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.738+0000 7fb29a7fc640 10 monclient: dump: 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:34.738 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:34.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.739+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fb29401ac80 con 0x7fb2a413f710 2024-03-25T19:45:34.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.740+0000 7fb2a9ed5640 10 monclient: get_auth_request con 0x7fb28806ddf0 auth_method 0 2024-03-25T19:45:34.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.740+0000 7fb29a7fc640 10 client.4423.objecter ms_handle_connect 0x7fb28806ddf0 2024-03-25T19:45:34.741 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.742+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fb294054930 con 0x7fb2a413f710 2024-03-25T19:45:34.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.742+0000 7fb29a7fc640 10 client.4423.objecter ms_dispatch 0x7fb2a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:34.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.742+0000 7fb29a7fc640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:34.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.742+0000 7fb29a7fc640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:34.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.743+0000 7fb29a7fc640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T19:45:34.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.746+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb294050270 con 0x7fb2a413f710 2024-03-25T19:45:34.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.746+0000 7fb29a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:34.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.746+0000 7fb29a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:34.965 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.966+0000 7fb2ac139640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:34.965 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.966+0000 7fb2ac139640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:34.965 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.966+0000 7fb2ac139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:45:34.965 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:34.966+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2a414f240 con 0x7fb2a413f710 2024-03-25T19:45:35.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.052+0000 7fb29a7fc640 1 -- 172.21.15.69:0/824578096 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fb294052370 con 0x7fb2a413f710 2024-03-25T19:45:35.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.052+0000 7fb29a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:35.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.052+0000 7fb29a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:35.052 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:45:35.052 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 10 librados: watch_flush enter 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 10 librados: watch_flush exit 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 10 client.4423.objecter successfully canceled tick 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 >> v1:172.21.15.169:6800/3072548259 conn(0x7fb28806ddf0 legacy=0x7fb2880702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 10 monclient: shutdown 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 >> v1:172.21.15.169:6789/0 conn(0x7fb2a413f710 legacy=0x7fb2a41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:35.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 shutdown_connections 2024-03-25T19:45:35.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 >> 172.21.15.69:0/824578096 conn(0x7fb2a40b7ea0 msgr2=0x7fb2a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:35.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 shutdown_connections 2024-03-25T19:45:35.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 1 -- 172.21.15.69:0/824578096 wait complete. 2024-03-25T19:45:35.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:35.055+0000 7fb2ac139640 1 librados: shutdown 2024-03-25T19:45:35.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:41.872 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:41.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.986+0000 7f51a77d6640 10 monclient: get_monmap_and_config 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.986+0000 7f51a77d6640 10 monclient: build_initial_monmap 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.986+0000 7f51a77d6640 10 monclient: monmap: 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:45:41.987731+0000 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:45:41.987731+0000 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:45:41.987 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:41.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 1 Processor -- start 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 1 -- start start 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 10 monclient: init 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 10 monclient(hunting): picked mon.noname-a con 0x7f51a00a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 10 monclient(hunting): picked mon.noname-b con 0x7f51a013f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 10 monclient(hunting): picked mon.noname-c con 0x7f51a01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a0142520 con 0x7f51a00a98f0 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a00b28a0 con 0x7f51a01430c0 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a00b2a30 con 0x7f51a013f710 2024-03-25T19:45:41.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:41.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.987+0000 7f51a77d6640 10 monclient(hunting): authenticate will time out at 5154.256559s 2024-03-25T19:45:41.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.993+0000 7f51a4d71640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f51a013f710 0x7f51a01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:54518/0 (socket says 172.21.15.69:54518) 2024-03-25T19:45:41.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.993+0000 7f51a4d71640 1 -- 172.21.15.69:0/3986770748 learned_addr learned my addr 172.21.15.69:0/3986770748 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:41.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.999+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689267472 0 0) 0x7f51a00b2a30 con 0x7f51a013f710 2024-03-25T19:45:41.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.999+0000 7f5197fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:41.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.999+0000 7f5197fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:41.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.999+0000 7f5197fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:41.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.999+0000 7f5197fff640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:45:41.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:41.999+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c003520 con 0x7f51a013f710 2024-03-25T19:45:42.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.004+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1769910540 0 0) 0x7f518c003520 con 0x7f51a013f710 2024-03-25T19:45:42.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.004+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 >> v1:172.21.15.169:6789/0 conn(0x7f51a01430c0 legacy=0x7f51a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.004+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 >> v1:172.21.15.69:6789/0 conn(0x7f51a00a98f0 legacy=0x7f51a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:42.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 1 monclient: found mon.noname-b 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a00b44c0 con 0x7f51a013f710 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5190001690 con 0x7f51a013f710 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f51a77d6640 5 monclient: authenticate success, global_id 4433 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f51a77d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f5197fff640 10 monclient: dump: 2024-03-25T19:45:42.006 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:42.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.005+0000 7f51a77d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:42.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.023+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5190001860 con 0x7f51a013f710 2024-03-25T19:45:42.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.023+0000 7f5197fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:42.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.023+0000 7f51a77d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.023+0000 7f51a77d6640 4 set_mon_vals no callback set 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.023+0000 7f5197fff640 1 -- 172.21.15.69:0/3986770748 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5190001cb0 con 0x7f51a013f710 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.023+0000 7f5197fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.023+0000 7f5197fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.023+0000 7f5197fff640 10 monclient: dump: 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.024 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.025 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.025 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:42.025 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:42.025 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:42.025 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:42.025 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.024+0000 7f51a77d6640 10 monclient: shutdown 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.024+0000 7f51a77d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.024+0000 7f51a77d6640 1 -- 172.21.15.69:0/3986770748 >> v1:172.21.15.69:6790/0 conn(0x7f51a013f710 legacy=0x7f51a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.024+0000 7f51a77d6640 1 -- 172.21.15.69:0/3986770748 shutdown_connections 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.024+0000 7f51a77d6640 1 -- 172.21.15.69:0/3986770748 >> 172.21.15.69:0/3986770748 conn(0x7f51a00b7ea0 msgr2=0x7f51a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.024+0000 7f51a77d6640 1 -- 172.21.15.69:0/3986770748 shutdown_connections 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.024+0000 7f51a77d6640 1 -- 172.21.15.69:0/3986770748 wait complete. 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.025+0000 7f51a77d6640 10 monclient: build_initial_monmap 2024-03-25T19:45:42.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.025+0000 7f51a77d6640 10 monclient: monmap: 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 1 librados: starting msgr at 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 1 librados: starting objecter 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 1 Processor -- start 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 1 -- start start 2024-03-25T19:45:42.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 1 librados: setting wanted keys 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 1 librados: calling monclient init 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 10 monclient: init 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.026+0000 7f51a77d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.027+0000 7f51a77d6640 10 monclient(hunting): picked mon.noname-a con 0x7f51a00a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.027+0000 7f51a77d6640 10 monclient(hunting): picked mon.noname-b con 0x7f51a013f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.027+0000 7f51a77d6640 10 monclient(hunting): picked mon.noname-c con 0x7f51a01430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.027+0000 7f51a77d6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a013aea0 con 0x7f51a00a98f0 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.027+0000 7f51a77d6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a0139f50 con 0x7f51a013f710 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.027+0000 7f51a77d6640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a013b0d0 con 0x7f51a01430c0 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.027+0000 7f51a77d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:42.030 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.027+0000 7f51a77d6640 10 monclient(hunting): authenticate will time out at 5154.295829s 2024-03-25T19:45:42.041 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.041+0000 7f51a5572640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f51a00a98f0 0x7f51a00b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52670/0 (socket says 172.21.15.69:52670) 2024-03-25T19:45:42.041 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.041+0000 7f51a5572640 1 -- 172.21.15.69:0/1155029095 learned_addr learned my addr 172.21.15.69:0/1155029095 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:42.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.042+0000 7f5195ffb640 10 client.?.objecter ms_handle_connect 0x7f51a00a98f0 2024-03-25T19:45:42.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.042+0000 7f5195ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:42.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.044+0000 7f5195ffb640 10 client.?.objecter ms_handle_connect 0x7f51a013f710 2024-03-25T19:45:42.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.044+0000 7f5195ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:42.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.050+0000 7f5195ffb640 10 client.?.objecter ms_handle_connect 0x7f51a01430c0 2024-03-25T19:45:42.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.050+0000 7f5195ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:42.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.057+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596174825 0 0) 0x7f51a013aea0 con 0x7f51a00a98f0 2024-03-25T19:45:42.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.057+0000 7f5195ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:42.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.057+0000 7f5195ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:42.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.057+0000 7f5195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:42.057 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.057+0000 7f5195ffb640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:45:42.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.057+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5184004eb0 con 0x7f51a00a98f0 2024-03-25T19:45:42.062 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.062+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609829844 0 0) 0x7f51a0139f50 con 0x7f51a013f710 2024-03-25T19:45:42.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.062+0000 7f5195ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:42.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.062+0000 7f5195ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:42.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.062+0000 7f5195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:42.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.062+0000 7f5195ffb640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:45:42.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.062+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51a013aea0 con 0x7f51a013f710 2024-03-25T19:45:42.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.064+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903930725 0 0) 0x7f51a013b0d0 con 0x7f51a01430c0 2024-03-25T19:45:42.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.064+0000 7f5195ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:42.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.064+0000 7f5195ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:42.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.064+0000 7f5195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:42.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.064+0000 7f5195ffb640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:45:42.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.064+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51a0139f50 con 0x7f51a01430c0 2024-03-25T19:45:42.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.075+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2260494781 0 0) 0x7f5184004eb0 con 0x7f51a00a98f0 2024-03-25T19:45:42.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.075+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51a013b0d0 con 0x7f51a00a98f0 2024-03-25T19:45:42.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.075+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f519c001960 con 0x7f51a00a98f0 2024-03-25T19:45:42.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.075+0000 7f5195ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.075+0000 7f5195ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.075+0000 7f5195ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:42.076 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:42.077 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:42.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.075+0000 7f5195ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:42.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.079+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 289219711 0 0) 0x7f51a013aea0 con 0x7f51a013f710 2024-03-25T19:45:42.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.079+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5184004eb0 con 0x7f51a013f710 2024-03-25T19:45:42.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.079+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5190001b70 con 0x7f51a013f710 2024-03-25T19:45:42.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.079+0000 7f5195ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.079+0000 7f5195ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.079+0000 7f5195ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:42.080 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1939308178 0 0) 0x7f51a013b0d0 con 0x7f51a00a98f0 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 >> v1:172.21.15.69:6790/0 conn(0x7f51a01430c0 legacy=0x7f51a013c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 >> v1:172.21.15.169:6789/0 conn(0x7f51a013f710 legacy=0x7f51a013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 1 monclient: found mon.a 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a013b3b0 con 0x7f51a00a98f0 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:42.085 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:42.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f5195ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:42.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 5 monclient: authenticate success, global_id 4449 2024-03-25T19:45:42.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 10 monclient: _renew_subs 2024-03-25T19:45:42.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:42.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a013b690 con 0x7f51a00a98f0 2024-03-25T19:45:42.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:42.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 10 monclient: _renew_subs 2024-03-25T19:45:42.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:42.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a013c8d0 con 0x7f51a00a98f0 2024-03-25T19:45:42.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.084+0000 7f51a77d6640 1 librados: init done 2024-03-25T19:45:42.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.085+0000 7f51a77d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:42.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.085+0000 7f51a77d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:42.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.085+0000 7f51a77d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:42.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.085+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a00b3ef0 con 0x7f51a00a98f0 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.417+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f519c0035f0 con 0x7f51a00a98f0 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.417+0000 7f5195ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.417+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f519c003940 con 0x7f51a00a98f0 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.417+0000 7f5195ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.417+0000 7f5195ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.417+0000 7f5195ffb640 10 monclient: dump: 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.417 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:42.418 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.418 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:42.418 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:42.418 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:42.418 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:42.418 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:42.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.417+0000 7f51977fe640 4 set_mon_vals no callback set 2024-03-25T19:45:42.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.417+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f519c003b60 con 0x7f51a00a98f0 2024-03-25T19:45:42.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.418+0000 7f51a4d71640 10 monclient: get_auth_request con 0x7f518406df70 auth_method 0 2024-03-25T19:45:42.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.418+0000 7f5195ffb640 10 client.4449.objecter ms_handle_connect 0x7f518406df70 2024-03-25T19:45:42.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.423+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f519c054750 con 0x7f51a00a98f0 2024-03-25T19:45:42.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.423+0000 7f5195ffb640 10 client.4449.objecter ms_dispatch 0x7f51a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:42.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.423+0000 7f5195ffb640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:42.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.423+0000 7f5195ffb640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:42.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.424+0000 7f5195ffb640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-25T19:45:42.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.424+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f519c050260 con 0x7f51a00a98f0 2024-03-25T19:45:42.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.424+0000 7f5195ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:42.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.424+0000 7f5195ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:42.504 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:20.468 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.565 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:20.529 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.610 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:20.564 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.612 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:20.576 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.617 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:20.571 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.630 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:20.594 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.636 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:20.601 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.638 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:20.592 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.641+0000 7f51a77d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:42.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.641+0000 7f51a77d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:42.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.641+0000 7f51a77d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:42.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.641+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51a00627b0 con 0x7f51a00a98f0 2024-03-25T19:45:42.661 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:20.624 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.715+0000 7f5195ffb640 1 -- 172.21.15.69:0/1155029095 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f519c052190 con 0x7f51a00a98f0 2024-03-25T19:45:42.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.715+0000 7f5195ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:42.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.715+0000 7f5195ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:42.715 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:45:42.716 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:42.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 10 librados: watch_flush enter 2024-03-25T19:45:42.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 10 librados: watch_flush exit 2024-03-25T19:45:42.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-25T19:45:42.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 10 client.4449.objecter successfully canceled tick 2024-03-25T19:45:42.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 >> v1:172.21.15.169:6800/3072548259 conn(0x7f518406df70 legacy=0x7f5184070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 10 monclient: shutdown 2024-03-25T19:45:42.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:42.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 >> v1:172.21.15.69:6789/0 conn(0x7f51a00a98f0 legacy=0x7f51a00b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 shutdown_connections 2024-03-25T19:45:42.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 >> 172.21.15.69:0/1155029095 conn(0x7f51a00b7ea0 msgr2=0x7f51a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:42.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 shutdown_connections 2024-03-25T19:45:42.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 1 -- 172.21.15.69:0/1155029095 wait complete. 2024-03-25T19:45:42.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:42.717+0000 7f51a77d6640 1 librados: shutdown 2024-03-25T19:45:42.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:49.537 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:49.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.640+0000 7f8c885af640 10 monclient: get_monmap_and_config 2024-03-25T19:45:49.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.640+0000 7f8c885af640 10 monclient: build_initial_monmap 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.640+0000 7f8c885af640 10 monclient: monmap: 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:45:49.641649+0000 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:45:49.641649+0000 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:45:49.641 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:49.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.641+0000 7f8c885af640 1 Processor -- start 2024-03-25T19:45:49.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.641+0000 7f8c885af640 1 -- start start 2024-03-25T19:45:49.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.641+0000 7f8c885af640 10 monclient: init 2024-03-25T19:45:49.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.641+0000 7f8c885af640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.641+0000 7f8c885af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.641+0000 7f8c885af640 10 monclient(hunting): picked mon.noname-b con 0x7f8c800a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.641+0000 7f8c885af640 10 monclient(hunting): picked mon.noname-c con 0x7f8c8013f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.642+0000 7f8c885af640 10 monclient(hunting): picked mon.noname-a con 0x7f8c801430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.642+0000 7f8c885af640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c80142520 con 0x7f8c801430c0 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.642+0000 7f8c885af640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c800b28a0 con 0x7f8c8013f710 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.642+0000 7f8c885af640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c800b2a30 con 0x7f8c800a98f0 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.642+0000 7f8c885af640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:49.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.642+0000 7f8c885af640 10 monclient(hunting): authenticate will time out at 5161.910674s 2024-03-25T19:45:49.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.654+0000 7f8c8634b640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f8c800a98f0 0x7f8c800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:59274/0 (socket says 172.21.15.69:59274) 2024-03-25T19:45:49.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.654+0000 7f8c8634b640 1 -- 172.21.15.69:0/3109419783 learned_addr learned my addr 172.21.15.69:0/3109419783 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:49.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.665+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103484149 0 0) 0x7f8c800b2a30 con 0x7f8c800a98f0 2024-03-25T19:45:49.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.665+0000 7f8c85349640 10 monclient(hunting): _negotiate 2024-03-25T19:45:49.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.665+0000 7f8c85349640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:49.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.665+0000 7f8c85349640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:49.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.665+0000 7f8c85349640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:45:49.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.665+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c6c003520 con 0x7f8c800a98f0 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 821239297 0 0) 0x7f8c6c003520 con 0x7f8c800a98f0 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 >> v1:172.21.15.169:6789/0 conn(0x7f8c8013f710 legacy=0x7f8c801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 >> v1:172.21.15.69:6789/0 conn(0x7f8c801430c0 legacy=0x7f8c8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 10 monclient: _finish_hunting 0 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 1 monclient: found mon.noname-b 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c800b44c0 con 0x7f8c800a98f0 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 10 monclient: _finish_auth 0 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 10 monclient: _check_auth_tickets 2024-03-25T19:45:49.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.673+0000 7f8c885af640 5 monclient: authenticate success, global_id 4439 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.673+0000 7f8c885af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.672+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8c7c001690 con 0x7f8c800a98f0 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.673+0000 7f8c85349640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.673+0000 7f8c85349640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.673+0000 7f8c85349640 10 monclient: dump: 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:49.674 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:49.675 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:49.675 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:49.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.673+0000 7f8c885af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:49.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.679+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c7c0017f0 con 0x7f8c800a98f0 2024-03-25T19:45:49.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.679+0000 7f8c85349640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:49.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.679+0000 7f8c85349640 1 -- 172.21.15.69:0/3109419783 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8c7c001c00 con 0x7f8c800a98f0 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.679+0000 7f8c85349640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.679+0000 7f8c85349640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.679+0000 7f8c85349640 10 monclient: dump: 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.679+0000 7f8c885af640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:49.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.679+0000 7f8c885af640 4 set_mon_vals no callback set 2024-03-25T19:45:49.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.680+0000 7f8c885af640 10 monclient: shutdown 2024-03-25T19:45:49.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.680+0000 7f8c885af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:49.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.680+0000 7f8c885af640 1 -- 172.21.15.69:0/3109419783 >> v1:172.21.15.69:6790/0 conn(0x7f8c800a98f0 legacy=0x7f8c800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.680+0000 7f8c885af640 1 -- 172.21.15.69:0/3109419783 shutdown_connections 2024-03-25T19:45:49.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.680+0000 7f8c885af640 1 -- 172.21.15.69:0/3109419783 >> 172.21.15.69:0/3109419783 conn(0x7f8c800b7ea0 msgr2=0x7f8c800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:49.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.681+0000 7f8c885af640 1 -- 172.21.15.69:0/3109419783 shutdown_connections 2024-03-25T19:45:49.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.681+0000 7f8c885af640 1 -- 172.21.15.69:0/3109419783 wait complete. 2024-03-25T19:45:49.682 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.681+0000 7f8c885af640 10 monclient: build_initial_monmap 2024-03-25T19:45:49.682 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.681+0000 7f8c885af640 10 monclient: monmap: 2024-03-25T19:45:49.682 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:49.682 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.682+0000 7f8c885af640 1 librados: starting msgr at 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.682+0000 7f8c885af640 1 librados: starting objecter 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.682+0000 7f8c885af640 1 Processor -- start 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.682+0000 7f8c885af640 1 -- start start 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.682+0000 7f8c885af640 1 librados: setting wanted keys 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.682+0000 7f8c885af640 1 librados: calling monclient init 2024-03-25T19:45:49.683 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.682+0000 7f8c885af640 10 monclient: init 2024-03-25T19:45:49.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:49.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:49.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 10 monclient(hunting): picked mon.noname-c con 0x7f8c800a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:49.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 10 monclient(hunting): picked mon.noname-a con 0x7f8c8013f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:49.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 10 monclient(hunting): picked mon.noname-b con 0x7f8c801430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:49.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c80135730 con 0x7f8c8013f710 2024-03-25T19:45:49.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c801358c0 con 0x7f8c801430c0 2024-03-25T19:45:49.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c80135a50 con 0x7f8c800a98f0 2024-03-25T19:45:49.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:49.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.683+0000 7f8c885af640 10 monclient(hunting): authenticate will time out at 5161.952043s 2024-03-25T19:45:49.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.691+0000 7f8c8634b640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f8c800a98f0 0x7f8c801dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:59324/0 (socket says 172.21.15.69:59324) 2024-03-25T19:45:49.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.691+0000 7f8c8634b640 1 -- 172.21.15.69:0/3091717978 learned_addr learned my addr 172.21.15.69:0/3091717978 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:49.693 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.693+0000 7f8c76ffd640 10 client.?.objecter ms_handle_connect 0x7f8c8013f710 2024-03-25T19:45:49.693 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.694+0000 7f8c76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:49.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.694+0000 7f8c76ffd640 10 client.?.objecter ms_handle_connect 0x7f8c800a98f0 2024-03-25T19:45:49.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.694+0000 7f8c76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:49.696 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.697+0000 7f8c76ffd640 10 client.?.objecter ms_handle_connect 0x7f8c801430c0 2024-03-25T19:45:49.697 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.697+0000 7f8c76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:49.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.702+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538944256 0 0) 0x7f8c80135730 con 0x7f8c8013f710 2024-03-25T19:45:49.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.702+0000 7f8c76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:49.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.702+0000 7f8c76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:49.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.702+0000 7f8c76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:49.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.702+0000 7f8c76ffd640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:45:49.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.702+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c64004eb0 con 0x7f8c8013f710 2024-03-25T19:45:49.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068120065 0 0) 0x7f8c801358c0 con 0x7f8c801430c0 2024-03-25T19:45:49.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:49.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:49.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:49.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:45:49.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c80135730 con 0x7f8c801430c0 2024-03-25T19:45:49.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345727932 0 0) 0x7f8c80135a50 con 0x7f8c800a98f0 2024-03-25T19:45:49.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:49.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:49.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:49.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:45:49.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.708+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c801358c0 con 0x7f8c800a98f0 2024-03-25T19:45:49.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.718+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426752848 0 0) 0x7f8c64004eb0 con 0x7f8c8013f710 2024-03-25T19:45:49.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.718+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c80135a50 con 0x7f8c8013f710 2024-03-25T19:45:49.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.718+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8c70001930 con 0x7f8c8013f710 2024-03-25T19:45:49.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.718+0000 7f8c76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:49.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.718+0000 7f8c76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:49.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.718+0000 7f8c76ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:49.718 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:49.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.718+0000 7f8c76ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1636409375 0 0) 0x7f8c80135a50 con 0x7f8c8013f710 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 >> v1:172.21.15.69:6790/0 conn(0x7f8c800a98f0 legacy=0x7f8c801dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 >> v1:172.21.15.169:6789/0 conn(0x7f8c801430c0 legacy=0x7f8c801d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 1 monclient: found mon.a 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c801d5000 con 0x7f8c8013f710 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:49.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 10 monclient: _renew_subs 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c801d52b0 con 0x7f8c8013f710 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 10 monclient: _renew_subs 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8013c900 con 0x7f8c8013f710 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.734+0000 7f8c885af640 1 librados: init done 2024-03-25T19:45:49.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.736+0000 7f8c885af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:49.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.736+0000 7f8c885af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:49.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.736+0000 7f8c885af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:49.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:49.736+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8013cbb0 con 0x7f8c8013f710 2024-03-25T19:45:50.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.059+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c70003610 con 0x7f8c8013f710 2024-03-25T19:45:50.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.059+0000 7f8c76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.059+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f8c70003880 con 0x7f8c8013f710 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.059+0000 7f8c76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.059+0000 7f8c76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.059+0000 7f8c76ffd640 10 monclient: dump: 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.059+0000 7f8c84b48640 4 set_mon_vals no callback set 2024-03-25T19:45:50.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.060+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f8c70003aa0 con 0x7f8c8013f710 2024-03-25T19:45:50.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.060+0000 7f8c8634b640 10 monclient: get_auth_request con 0x7f8c6406ddf0 auth_method 0 2024-03-25T19:45:50.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.060+0000 7f8c76ffd640 10 client.4455.objecter ms_handle_connect 0x7f8c6406ddf0 2024-03-25T19:45:50.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.068+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f8c70053470 con 0x7f8c8013f710 2024-03-25T19:45:50.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.068+0000 7f8c76ffd640 10 client.4455.objecter ms_dispatch 0x7f8c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:50.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.068+0000 7f8c76ffd640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:50.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.068+0000 7f8c76ffd640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:50.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.068+0000 7f8c76ffd640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T19:45:50.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.068+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c7004ef80 con 0x7f8c8013f710 2024-03-25T19:45:50.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.068+0000 7f8c76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:50.068 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.068+0000 7f8c76ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:50.318 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.318+0000 7f8c885af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:50.318 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.318+0000 7f8c885af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:50.318 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.318+0000 7f8c885af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:50.319 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.318+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c801371c0 con 0x7f8c8013f710 2024-03-25T19:45:50.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.342+0000 7f8c76ffd640 1 -- 172.21.15.69:0/3091717978 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f8c70050eb0 con 0x7f8c8013f710 2024-03-25T19:45:50.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.342+0000 7f8c76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:50.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.342+0000 7f8c76ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:50.342 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:45:50.343 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:50.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.344+0000 7f8c885af640 10 librados: watch_flush enter 2024-03-25T19:45:50.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.344+0000 7f8c885af640 10 librados: watch_flush exit 2024-03-25T19:45:50.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.344+0000 7f8c885af640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T19:45:50.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.344+0000 7f8c885af640 10 client.4455.objecter successfully canceled tick 2024-03-25T19:45:50.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 >> v1:172.21.15.169:6800/3072548259 conn(0x7f8c6406ddf0 legacy=0x7f8c640702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:50.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 10 monclient: shutdown 2024-03-25T19:45:50.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:50.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 >> v1:172.21.15.69:6789/0 conn(0x7f8c8013f710 legacy=0x7f8c801d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:50.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 shutdown_connections 2024-03-25T19:45:50.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 >> 172.21.15.69:0/3091717978 conn(0x7f8c800b7ea0 msgr2=0x7f8c8014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:50.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 shutdown_connections 2024-03-25T19:45:50.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 1 -- 172.21.15.69:0/3091717978 wait complete. 2024-03-25T19:45:50.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:50.345+0000 7f8c885af640 1 librados: shutdown 2024-03-25T19:45:50.356 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:57.164 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:57.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.278+0000 7fc67fb9a640 10 monclient: get_monmap_and_config 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.278+0000 7fc67fb9a640 10 monclient: build_initial_monmap 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.278+0000 7fc67fb9a640 10 monclient: monmap: 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:45:57.279844+0000 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:45:57.279844+0000 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:45:57.279 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:57.280 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.279+0000 7fc67fb9a640 1 Processor -- start 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.279+0000 7fc67fb9a640 1 -- start start 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.279+0000 7fc67fb9a640 10 monclient: init 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 10 monclient(hunting): picked mon.noname-a con 0x7fc6780b4f10 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 10 monclient(hunting): picked mon.noname-c con 0x7fc6780b44d0 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 10 monclient(hunting): picked mon.noname-b con 0x7fc67813f740 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6780b0140 con 0x7fc6780b4f10 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6780af3a0 con 0x7fc6780b44d0 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6780af530 con 0x7fc67813f740 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:57.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.280+0000 7fc67fb9a640 10 monclient(hunting): authenticate will time out at 5169.548998s 2024-03-25T19:45:57.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.292+0000 7fc67d135640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc6780b44d0 0x7fc6780b16c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37876/0 (socket says 172.21.15.69:37876) 2024-03-25T19:45:57.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.292+0000 7fc67d135640 1 -- 172.21.15.69:0/614648076 learned_addr learned my addr 172.21.15.69:0/614648076 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:57.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.304+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401215670 0 0) 0x7fc6780af3a0 con 0x7fc6780b44d0 2024-03-25T19:45:57.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.304+0000 7fc67c934640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.304+0000 7fc67c934640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.304+0000 7fc67c934640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.304+0000 7fc67c934640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:45:57.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.304+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc664003520 con 0x7fc6780b44d0 2024-03-25T19:45:57.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.315+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192230811 0 0) 0x7fc6780b0140 con 0x7fc6780b4f10 2024-03-25T19:45:57.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.315+0000 7fc67c934640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.315+0000 7fc67c934640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.315+0000 7fc67c934640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.315+0000 7fc67c934640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.315+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6780af3a0 con 0x7fc6780b4f10 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 38278542 0 0) 0x7fc664003520 con 0x7fc6780b44d0 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 >> v1:172.21.15.69:6790/0 conn(0x7fc67813f740 legacy=0x7fc67813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 >> v1:172.21.15.69:6789/0 conn(0x7fc6780b4f10 legacy=0x7fc6780ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 10 monclient: _finish_hunting 0 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 1 monclient: found mon.noname-c 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780afcd0 con 0x7fc6780b44d0 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 10 monclient: _finish_auth 0 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 10 monclient: _check_auth_tickets 2024-03-25T19:45:57.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc668001690 con 0x7fc6780b44d0 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67c934640 10 monclient: dump: 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:57.318 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:57.319 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:57.319 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67fb9a640 5 monclient: authenticate success, global_id 4438 2024-03-25T19:45:57.319 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.316+0000 7fc67fb9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:57.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6680017f0 con 0x7fc6780b44d0 2024-03-25T19:45:57.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67c934640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:57.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67c934640 1 -- 172.21.15.69:0/614648076 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc668001c00 con 0x7fc6780b44d0 2024-03-25T19:45:57.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67fb9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:57.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67fb9a640 4 set_mon_vals no callback set 2024-03-25T19:45:57.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67c934640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67c934640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:57.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67c934640 10 monclient: dump: 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:57.334 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:57.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67fb9a640 10 monclient: shutdown 2024-03-25T19:45:57.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67fb9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.333+0000 7fc67fb9a640 1 -- 172.21.15.69:0/614648076 >> v1:172.21.15.169:6789/0 conn(0x7fc6780b44d0 legacy=0x7fc6780b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.334+0000 7fc67fb9a640 1 -- 172.21.15.69:0/614648076 shutdown_connections 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.334+0000 7fc67fb9a640 1 -- 172.21.15.69:0/614648076 >> 172.21.15.69:0/614648076 conn(0x7fc6780617a0 msgr2=0x7fc678061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.334+0000 7fc67fb9a640 1 -- 172.21.15.69:0/614648076 shutdown_connections 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.334+0000 7fc67fb9a640 1 -- 172.21.15.69:0/614648076 wait complete. 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 10 monclient: build_initial_monmap 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 10 monclient: monmap: 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:57.337 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 1 librados: starting msgr at 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 1 librados: starting objecter 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 1 Processor -- start 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 1 -- start start 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 1 librados: setting wanted keys 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 1 librados: calling monclient init 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.335+0000 7fc67fb9a640 10 monclient: init 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:57.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:57.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 10 monclient(hunting): picked mon.noname-c con 0x7fc6780b44d0 addr v1:172.21.15.69:6790/0 2024-03-25T19:45:57.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 10 monclient(hunting): picked mon.noname-a con 0x7fc6780b4f10 addr v1:172.21.15.69:6789/0 2024-03-25T19:45:57.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 10 monclient(hunting): picked mon.noname-b con 0x7fc67813f740 addr v1:172.21.15.169:6789/0 2024-03-25T19:45:57.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6781d7710 con 0x7fc6780b4f10 2024-03-25T19:45:57.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6781d79f0 con 0x7fc67813f740 2024-03-25T19:45:57.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6780a9470 con 0x7fc6780b44d0 2024-03-25T19:45:57.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:57.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.336+0000 7fc67fb9a640 10 monclient(hunting): authenticate will time out at 5169.604935s 2024-03-25T19:45:57.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.343+0000 7fc67e137640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc67813f740 0x7fc6781d49c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37900/0 (socket says 172.21.15.69:37900) 2024-03-25T19:45:57.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.343+0000 7fc67e137640 1 -- 172.21.15.69:0/3607059739 learned_addr learned my addr 172.21.15.69:0/3607059739 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:45:57.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.347+0000 7fc67e137640 1 -- 172.21.15.69:0/3607059739 >> v1:172.21.15.169:6789/0 conn(0x7fc67813f740 legacy=0x7fc6781d49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:45:57.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.347+0000 7fc67e137640 1 -- 172.21.15.69:0/3607059739 >> v1:172.21.15.169:6789/0 conn(0x7fc67813f740 legacy=0x7fc6781d49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:45:57.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.347+0000 7fc67e137640 1 --1- 172.21.15.69:0/3607059739 >> v1:172.21.15.169:6789/0 conn(0x7fc67813f740 0x7fc6781d49c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T19:45:57.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.347+0000 7fc67e137640 1 --1- 172.21.15.69:0/3607059739 >> v1:172.21.15.169:6789/0 conn(0x7fc67813f740 0x7fc6781d49c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T19:45:57.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.347+0000 7fc66e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.169:6789/0 2024-03-25T19:45:57.359 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.359+0000 7fc66e7fc640 10 client.?.objecter ms_handle_connect 0x7fc6780b4f10 2024-03-25T19:45:57.359 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.359+0000 7fc66e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:57.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.360+0000 7fc66e7fc640 10 client.?.objecter ms_handle_connect 0x7fc6780b44d0 2024-03-25T19:45:57.360 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.360+0000 7fc66e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:57.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.372+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52266775 0 0) 0x7fc6781d7710 con 0x7fc6780b4f10 2024-03-25T19:45:57.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.372+0000 7fc66e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.372+0000 7fc66e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.372+0000 7fc66e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.372+0000 7fc66e7fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:45:57.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.373+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65c004eb0 con 0x7fc6780b4f10 2024-03-25T19:45:57.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.373+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951759983 0 0) 0x7fc6780a9470 con 0x7fc6780b44d0 2024-03-25T19:45:57.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.373+0000 7fc66e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.373+0000 7fc66e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.373+0000 7fc66e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.373+0000 7fc66e7fc640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:45:57.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.373+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6781d7710 con 0x7fc6780b44d0 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.394+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4187989781 0 0) 0x7fc6781d7710 con 0x7fc6780b44d0 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.394+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6780a9470 con 0x7fc6780b44d0 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.394+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc674001930 con 0x7fc6780b44d0 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.394+0000 7fc66e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.394+0000 7fc66e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.394+0000 7fc66e7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.395 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:57.396 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:57.396 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:57.396 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:57.396 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:57.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.394+0000 7fc66e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:57.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.395+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3615389060 0 0) 0x7fc65c004eb0 con 0x7fc6780b4f10 2024-03-25T19:45:57.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.395+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6781d7710 con 0x7fc6780b4f10 2024-03-25T19:45:57.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.395+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc668001b00 con 0x7fc6780b4f10 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.395+0000 7fc66e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.395+0000 7fc66e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.395+0000 7fc66e7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:57.397 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:57.398 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:57.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 77081217 0 0) 0x7fc6781d7710 con 0x7fc6780b4f10 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 >> v1:172.21.15.69:6790/0 conn(0x7fc6780b44d0 legacy=0x7fc6781d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 >> v1:172.21.15.169:6789/0 conn(0x7fc67813f740 legacy=0x7fc6781d49c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 1 monclient: found mon.a 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6781358b0 con 0x7fc6780b4f10 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc66e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 5 monclient: authenticate success, global_id 4461 2024-03-25T19:45:57.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 10 monclient: _renew_subs 2024-03-25T19:45:57.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:57.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6781d50d0 con 0x7fc6780b4f10 2024-03-25T19:45:57.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:57.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 10 monclient: _renew_subs 2024-03-25T19:45:57.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:57.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6780a9e50 con 0x7fc6780b4f10 2024-03-25T19:45:57.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.404+0000 7fc67fb9a640 1 librados: init done 2024-03-25T19:45:57.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.405+0000 7fc67fb9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:57.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.405+0000 7fc67fb9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:57.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.405+0000 7fc67fb9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:57.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.405+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6781d7b80 con 0x7fc6780b4f10 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.716+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc668004a90 con 0x7fc6780b4f10 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.716+0000 7fc66e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.716+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc668004da0 con 0x7fc6780b4f10 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.716+0000 7fc66e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.716+0000 7fc66e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.716+0000 7fc66e7fc640 10 monclient: dump: 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:45:57.717 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.718 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:45:57.718 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:45:57.718 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:45:57.718 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:45:57.718 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:45:57.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.716+0000 7fc66ffff640 4 set_mon_vals no callback set 2024-03-25T19:45:57.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.717+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fc66801ad70 con 0x7fc6780b4f10 2024-03-25T19:45:57.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.718+0000 7fc67d936640 10 monclient: get_auth_request con 0x7fc65c06de90 auth_method 0 2024-03-25T19:45:57.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.718+0000 7fc66e7fc640 10 client.4461.objecter ms_handle_connect 0x7fc65c06de90 2024-03-25T19:45:57.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.724+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fc6680548e0 con 0x7fc6780b4f10 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.724+0000 7fc66e7fc640 10 client.4461.objecter ms_dispatch 0x7fc678000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.724+0000 7fc66e7fc640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.724+0000 7fc66e7fc640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.724+0000 7fc66e7fc640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.724+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6680503f0 con 0x7fc6780b4f10 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.724+0000 7fc66e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.724+0000 7fc66e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:57.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.981+0000 7fc67fb9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:57.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.981+0000 7fc67fb9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:57.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.981+0000 7fc67fb9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:45:57.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:57.981+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6781d5260 con 0x7fc6780b4f10 2024-03-25T19:45:58.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.042+0000 7fc66e7fc640 1 -- 172.21.15.69:0/3607059739 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fc668052320 con 0x7fc6780b4f10 2024-03-25T19:45:58.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.042+0000 7fc66e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:58.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.042+0000 7fc66e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:58.043 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:45:58.043 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:58.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 10 librados: watch_flush enter 2024-03-25T19:45:58.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 10 librados: watch_flush exit 2024-03-25T19:45:58.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T19:45:58.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 10 client.4461.objecter successfully canceled tick 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 >> v1:172.21.15.169:6800/3072548259 conn(0x7fc65c06de90 legacy=0x7fc65c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 10 monclient: shutdown 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 >> v1:172.21.15.69:6789/0 conn(0x7fc6780b4f10 legacy=0x7fc6781351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 shutdown_connections 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 >> 172.21.15.69:0/3607059739 conn(0x7fc6780617a0 msgr2=0x7fc6780ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 shutdown_connections 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 1 -- 172.21.15.69:0/3607059739 wait complete. 2024-03-25T19:45:58.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:45:58.045+0000 7fc67fb9a640 1 librados: shutdown 2024-03-25T19:45:58.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:02.818 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:40.782 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.880 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:40.844 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.912 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:40.876 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.934 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:40.888 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.938 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:40.902 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.940 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:40.894 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.949 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:40.914 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.961 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:40.915 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.972 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:40.936 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.867 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:04.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.983+0000 7f7c11825640 10 monclient: get_monmap_and_config 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.983+0000 7f7c11825640 10 monclient: build_initial_monmap 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.983+0000 7f7c11825640 10 monclient: monmap: 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:46:04.984888+0000 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:46:04.984888+0000 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:46:04.984 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:04.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.984+0000 7f7c11825640 1 Processor -- start 2024-03-25T19:46:04.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.984+0000 7f7c11825640 1 -- start start 2024-03-25T19:46:04.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.984+0000 7f7c11825640 10 monclient: init 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 10 monclient(hunting): picked mon.noname-c con 0x7f7c0c0ad880 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0c0adc60 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c072200 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c072690 con 0x7f7c0c072200 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c06fc10 con 0x7f7c0c0ad880 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c0728d0 con 0x7f7c0c0adc60 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:04.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.985+0000 7f7c11825640 10 monclient(hunting): authenticate will time out at 5177.254226s 2024-03-25T19:46:04.998 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.998+0000 7f7c0b7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7c0c072200 0x7f7c0c13f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56368/0 (socket says 172.21.15.69:56368) 2024-03-25T19:46:04.998 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:04.998+0000 7f7c0b7fe640 1 -- 172.21.15.69:0/696777210 learned_addr learned my addr 172.21.15.69:0/696777210 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:05.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.010+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830848168 0 0) 0x7f7c0c0728d0 con 0x7f7c0c0adc60 2024-03-25T19:46:05.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.010+0000 7f7c0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.010+0000 7f7c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.010+0000 7f7c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.010+0000 7f7c0a7fc640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:46:05.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.010+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf8003520 con 0x7f7c0c0adc60 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930636977 0 0) 0x7f7c04001490 con 0x7f7c0c072200 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c0c0728d0 con 0x7f7c0c072200 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247888736 0 0) 0x7f7c0c06fc10 con 0x7f7c0c0ad880 2024-03-25T19:46:05.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:46:05.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.015+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c04001490 con 0x7f7c0c0ad880 2024-03-25T19:46:05.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3939825853 0 0) 0x7f7c04001490 con 0x7f7c0c0ad880 2024-03-25T19:46:05.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 >> v1:172.21.15.69:6790/0 conn(0x7f7c0c0adc60 legacy=0x7f7c0c06df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 >> v1:172.21.15.69:6789/0 conn(0x7f7c0c072200 legacy=0x7f7c0c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:05.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:46:05.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:05.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:46:05.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c0b02e0 con 0x7f7c0c0ad880 2024-03-25T19:46:05.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:05.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:05.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:05.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7bfc0015b0 con 0x7f7c0c0ad880 2024-03-25T19:46:05.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c11825640 5 monclient: authenticate success, global_id 4441 2024-03-25T19:46:05.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c11825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:05.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c0a7fc640 10 monclient: dump: 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:05.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.025+0000 7f7c11825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.054+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bfc001710 con 0x7f7c0c0ad880 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.054+0000 7f7c0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.054+0000 7f7c0a7fc640 1 -- 172.21.15.69:0/696777210 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7bfc001b60 con 0x7f7c0c0ad880 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.054+0000 7f7c11825640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.054+0000 7f7c11825640 4 set_mon_vals no callback set 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.054+0000 7f7c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.054+0000 7f7c0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.054+0000 7f7c0a7fc640 10 monclient: dump: 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:05.055 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:05.056 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:05.056 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:05.056 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.056 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:05.056 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:05.056 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:05.056 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:05.056 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.055+0000 7f7c11825640 10 monclient: shutdown 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.055+0000 7f7c11825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.055+0000 7f7c11825640 1 -- 172.21.15.69:0/696777210 >> v1:172.21.15.169:6789/0 conn(0x7f7c0c0ad880 legacy=0x7f7c0c1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.055+0000 7f7c11825640 1 -- 172.21.15.69:0/696777210 shutdown_connections 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.055+0000 7f7c11825640 1 -- 172.21.15.69:0/696777210 >> 172.21.15.69:0/696777210 conn(0x7f7c0c06b630 msgr2=0x7f7c0c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.055+0000 7f7c11825640 1 -- 172.21.15.69:0/696777210 shutdown_connections 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.055+0000 7f7c11825640 1 -- 172.21.15.69:0/696777210 wait complete. 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.056+0000 7f7c11825640 10 monclient: build_initial_monmap 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.056+0000 7f7c11825640 10 monclient: monmap: 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:05.058 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.057+0000 7f7c11825640 1 librados: starting msgr at 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.057+0000 7f7c11825640 1 librados: starting objecter 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.057+0000 7f7c11825640 1 Processor -- start 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.057+0000 7f7c11825640 1 -- start start 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.057+0000 7f7c11825640 1 librados: setting wanted keys 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.057+0000 7f7c11825640 1 librados: calling monclient init 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.057+0000 7f7c11825640 10 monclient: init 2024-03-25T19:46:05.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.057+0000 7f7c11825640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c072200 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0c0ad880 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 10 monclient(hunting): picked mon.noname-c con 0x7f7c0c13b110 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c13e340 con 0x7f7c0c072200 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c135670 con 0x7f7c0c0ad880 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c143b70 con 0x7f7c0c13b110 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:05.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.058+0000 7f7c11825640 10 monclient(hunting): authenticate will time out at 5177.326837s 2024-03-25T19:46:05.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.065+0000 7f7c0b7fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f7c0c13b110 0x7f7c0c134f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:35144/0 (socket says 172.21.15.69:35144) 2024-03-25T19:46:05.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.065+0000 7f7c0b7fe640 1 -- 172.21.15.69:0/1560306640 learned_addr learned my addr 172.21.15.69:0/1560306640 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:05.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.065+0000 7f7c08ff9640 10 client.?.objecter ms_handle_connect 0x7f7c0c13b110 2024-03-25T19:46:05.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.065+0000 7f7c08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:05.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.074+0000 7f7c08ff9640 10 client.?.objecter ms_handle_connect 0x7f7c0c072200 2024-03-25T19:46:05.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.074+0000 7f7c08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:05.078 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.078+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219532408 0 0) 0x7f7c0c143b70 con 0x7f7c0c13b110 2024-03-25T19:46:05.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.078+0000 7f7c08ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.078+0000 7f7c08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.078+0000 7f7c08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.078+0000 7f7c08ff9640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:46:05.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.078+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec004eb0 con 0x7f7c0c13b110 2024-03-25T19:46:05.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.079+0000 7f7c08ff9640 10 client.?.objecter ms_handle_connect 0x7f7c0c0ad880 2024-03-25T19:46:05.079 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.079+0000 7f7c08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:05.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.086+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776998402 0 0) 0x7f7c0c13e340 con 0x7f7c0c072200 2024-03-25T19:46:05.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.086+0000 7f7c08ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.086+0000 7f7c08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.086+0000 7f7c08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.086+0000 7f7c08ff9640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:46:05.086 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.086+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c0c143b70 con 0x7f7c0c072200 2024-03-25T19:46:05.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.095+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1968578406 0 0) 0x7f7bec004eb0 con 0x7f7c0c13b110 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.095+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c0c13e340 con 0x7f7c0c13b110 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.095+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7c04001ca0 con 0x7f7c0c13b110 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.095+0000 7f7c08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.095+0000 7f7c08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.095+0000 7f7c08ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:05.096 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:05.097 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:05.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.095+0000 7f7c08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:05.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.099+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864236064 0 0) 0x7f7c0c135670 con 0x7f7c0c0ad880 2024-03-25T19:46:05.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.099+0000 7f7c08ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.099+0000 7f7c08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.099+0000 7f7c08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.099+0000 7f7c08ff9640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:46:05.099 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.099+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec004eb0 con 0x7f7c0c0ad880 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 246641582 0 0) 0x7f7c0c13e340 con 0x7f7c0c13b110 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 >> v1:172.21.15.169:6789/0 conn(0x7f7c0c0ad880 legacy=0x7f7c0c134850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 >> v1:172.21.15.69:6789/0 conn(0x7f7c0c072200 legacy=0x7f7c0c13aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 1 monclient: found mon.c 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c143e50 con 0x7f7c0c13b110 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:05.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 5 monclient: authenticate success, global_id 4454 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 10 monclient: _renew_subs 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c13e7d0 con 0x7f7c0c13b110 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 10 monclient: _renew_subs 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c13de90 con 0x7f7c0c13b110 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.105+0000 7f7c11825640 1 librados: init done 2024-03-25T19:46:05.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.106+0000 7f7c11825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:05.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.106+0000 7f7c11825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:05.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.106+0000 7f7c11825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:05.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.106+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0c0627b0 con 0x7f7c0c13b110 2024-03-25T19:46:05.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.418+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c040020b0 con 0x7f7c0c13b110 2024-03-25T19:46:05.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.418+0000 7f7c08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.418+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7c04004d00 con 0x7f7c0c13b110 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.418+0000 7f7c08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.418+0000 7f7c08ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.418+0000 7f7c08ff9640 10 monclient: dump: 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.418+0000 7f7c09ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.419+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f7c04004f20 con 0x7f7c0c13b110 2024-03-25T19:46:05.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.419+0000 7f7c0affd640 10 monclient: get_auth_request con 0x7f7bec06de20 auth_method 0 2024-03-25T19:46:05.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.420+0000 7f7c08ff9640 10 client.4454.objecter ms_handle_connect 0x7f7bec06de20 2024-03-25T19:46:05.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.424+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f7c040549f0 con 0x7f7c0c13b110 2024-03-25T19:46:05.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.424+0000 7f7c08ff9640 10 client.4454.objecter ms_dispatch 0x7f7c0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:05.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.424+0000 7f7c08ff9640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:05.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.424+0000 7f7c08ff9640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:05.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.424+0000 7f7c08ff9640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T19:46:05.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.425+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c04050500 con 0x7f7c0c13b110 2024-03-25T19:46:05.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.425+0000 7f7c08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:05.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.425+0000 7f7c08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:05.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.674+0000 7f7c11825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:05.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.674+0000 7f7c11825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:05.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.674+0000 7f7c11825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:05.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.674+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c0c0aed80 con 0x7f7c0c13b110 2024-03-25T19:46:05.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.705+0000 7f7c08ff9640 1 -- 172.21.15.69:0/1560306640 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f7c04052430 con 0x7f7c0c13b110 2024-03-25T19:46:05.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.706+0000 7f7c08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:05.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.706+0000 7f7c08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:05.706 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:46:05.706 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:05.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.708+0000 7f7c11825640 10 librados: watch_flush enter 2024-03-25T19:46:05.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.708+0000 7f7c11825640 10 librados: watch_flush exit 2024-03-25T19:46:05.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.708+0000 7f7c11825640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T19:46:05.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.708+0000 7f7c11825640 10 client.4454.objecter successfully canceled tick 2024-03-25T19:46:05.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 >> v1:172.21.15.169:6800/3072548259 conn(0x7f7bec06de20 legacy=0x7f7bec0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 10 monclient: shutdown 2024-03-25T19:46:05.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:05.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 >> v1:172.21.15.69:6790/0 conn(0x7f7c0c13b110 legacy=0x7f7c0c134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 shutdown_connections 2024-03-25T19:46:05.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 >> 172.21.15.69:0/1560306640 conn(0x7f7c0c06b630 msgr2=0x7f7c0c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:05.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 shutdown_connections 2024-03-25T19:46:05.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 1 -- 172.21.15.69:0/1560306640 wait complete. 2024-03-25T19:46:05.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:05.709+0000 7f7c11825640 1 librados: shutdown 2024-03-25T19:46:05.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:12.528 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:12.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.641+0000 7fe8c9f9d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.641+0000 7fe8c9f9d640 10 monclient: build_initial_monmap 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.641+0000 7fe8c9f9d640 10 monclient: monmap: 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:46:12.642843+0000 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:46:12.642843+0000 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:12.642 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:46:12.643 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:46:12.643 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:12.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.642+0000 7fe8c9f9d640 1 Processor -- start 2024-03-25T19:46:12.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.642+0000 7fe8c9f9d640 1 -- start start 2024-03-25T19:46:12.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.642+0000 7fe8c9f9d640 10 monclient: init 2024-03-25T19:46:12.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:12.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:12.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c40b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8c40b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8c414be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c40af940 con 0x7fe8c40b4d60 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c40b0a70 con 0x7fe8c40b1730 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c40b0d50 con 0x7fe8c414be90 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.643+0000 7fe8c9f9d640 10 monclient(hunting): authenticate will time out at 5184.911742s 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.644+0000 7fe8c2ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe8c40b1730 0x7fe8c40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:60586/0 (socket says 172.21.15.69:60586) 2024-03-25T19:46:12.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.644+0000 7fe8c2ffd640 1 -- 172.21.15.69:0/3650216616 learned_addr learned my addr 172.21.15.69:0/3650216616 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:12.709 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.709+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875525613 0 0) 0x7fe8c40af940 con 0x7fe8c40b4d60 2024-03-25T19:46:12.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.709+0000 7fe8c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:12.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.709+0000 7fe8c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:12.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.709+0000 7fe8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:12.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.709+0000 7fe8c27fc640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:46:12.710 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.709+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ac003520 con 0x7fe8c40b4d60 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2673462722 0 0) 0x7fe8ac003520 con 0x7fe8c40b4d60 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 >> v1:172.21.15.69:6790/0 conn(0x7fe8c414be90 legacy=0x7fe8c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 >> v1:172.21.15.169:6789/0 conn(0x7fe8c40b1730 legacy=0x7fe8c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 1 monclient: found mon.noname-a 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c40b1030 con 0x7fe8c40b4d60 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:12.727 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe8b4001690 con 0x7fe8c40b4d60 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c27fc640 10 monclient: dump: 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:12.728 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:12.729 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:12.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.726+0000 7fe8c9f9d640 5 monclient: authenticate success, global_id 4470 2024-03-25T19:46:12.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.727+0000 7fe8c9f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:12.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.751+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8b40017f0 con 0x7fe8c40b4d60 2024-03-25T19:46:12.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.751+0000 7fe8c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:12.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.751+0000 7fe8c27fc640 1 -- 172.21.15.69:0/3650216616 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe8b4001c00 con 0x7fe8c40b4d60 2024-03-25T19:46:12.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.751+0000 7fe8c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.751+0000 7fe8c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:12.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.751+0000 7fe8c27fc640 10 monclient: dump: 2024-03-25T19:46:12.752 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:12.752 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:12.753 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:12.753 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:12.753 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.753 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:12.753 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:12.754 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:12.754 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:12.754 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:12.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.751+0000 7fe8c9f9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:12.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.751+0000 7fe8c9f9d640 4 set_mon_vals no callback set 2024-03-25T19:46:12.756 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.752+0000 7fe8c9f9d640 10 monclient: shutdown 2024-03-25T19:46:12.756 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.752+0000 7fe8c9f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.752+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/3650216616 >> v1:172.21.15.69:6789/0 conn(0x7fe8c40b4d60 legacy=0x7fe8c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.752+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/3650216616 shutdown_connections 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.752+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/3650216616 >> 172.21.15.69:0/3650216616 conn(0x7fe8c40617a0 msgr2=0x7fe8c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.752+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/3650216616 shutdown_connections 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.752+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/3650216616 wait complete. 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 10 monclient: build_initial_monmap 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 10 monclient: monmap: 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:12.757 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 1 librados: starting msgr at 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 1 librados: starting objecter 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 1 Processor -- start 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 1 -- start start 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 1 librados: setting wanted keys 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 1 librados: calling monclient init 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.753+0000 7fe8c9f9d640 10 monclient: init 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:12.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8c40b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:12.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8c40b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:12.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c414be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:12.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c413a940 con 0x7fe8c414be90 2024-03-25T19:46:12.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c413ab90 con 0x7fe8c40b4d60 2024-03-25T19:46:12.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c41d7840 con 0x7fe8c40b1730 2024-03-25T19:46:12.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:12.759 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.754+0000 7fe8c9f9d640 10 monclient(hunting): authenticate will time out at 5185.023141s 2024-03-25T19:46:12.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.776+0000 7fe8c3fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe8c414be90 0x7fe8c41e3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35900/0 (socket says 172.21.15.69:35900) 2024-03-25T19:46:12.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.776+0000 7fe8c3fff640 1 -- 172.21.15.69:0/4134135000 learned_addr learned my addr 172.21.15.69:0/4134135000 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:12.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.778+0000 7fe8abfff640 10 client.?.objecter ms_handle_connect 0x7fe8c414be90 2024-03-25T19:46:12.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.778+0000 7fe8abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:12.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.792+0000 7fe8abfff640 10 client.?.objecter ms_handle_connect 0x7fe8c40b4d60 2024-03-25T19:46:12.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.792+0000 7fe8abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:12.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.792+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760481059 0 0) 0x7fe8c413a940 con 0x7fe8c414be90 2024-03-25T19:46:12.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.792+0000 7fe8abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:12.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.792+0000 7fe8abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:12.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.792+0000 7fe8abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:12.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.792+0000 7fe8abfff640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:46:12.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.792+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b0004eb0 con 0x7fe8c414be90 2024-03-25T19:46:12.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.796+0000 7fe8abfff640 10 client.?.objecter ms_handle_connect 0x7fe8c40b1730 2024-03-25T19:46:12.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.796+0000 7fe8abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:12.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.822+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314816220 0 0) 0x7fe8b0004eb0 con 0x7fe8c414be90 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.822+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8c413a940 con 0x7fe8c414be90 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.822+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe8bc001c30 con 0x7fe8c414be90 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.822+0000 7fe8abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.822+0000 7fe8abfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.822+0000 7fe8abfff640 10 monclient(hunting): dump: 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.823 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:12.824 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:12.824 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:12.824 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:12.824 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:12.824 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.822+0000 7fe8abfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:12.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 124531345 0 0) 0x7fe8c413a940 con 0x7fe8c414be90 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 >> v1:172.21.15.69:6790/0 conn(0x7fe8c40b1730 legacy=0x7fe8c413bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 >> v1:172.21.15.169:6789/0 conn(0x7fe8c40b4d60 legacy=0x7fe8c41d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 1 monclient: found mon.a 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c413c240 con 0x7fe8c414be90 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 10 monclient: _finish_auth 0 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 5 monclient: authenticate success, global_id 4473 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 10 monclient: _renew_subs 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c4139ce0 con 0x7fe8c414be90 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 10 monclient: _renew_subs 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c41d8650 con 0x7fe8c414be90 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.876+0000 7fe8c9f9d640 1 librados: init done 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.877+0000 7fe8c9f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.877+0000 7fe8c9f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.877+0000 7fe8c9f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:12.877+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c41401f0 con 0x7fe8c414be90 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.162+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8bc003970 con 0x7fe8c414be90 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.162+0000 7fe8abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.162+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fe8bc003d40 con 0x7fe8c414be90 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.162+0000 7fe8abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.162+0000 7fe8abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.162+0000 7fe8abfff640 10 monclient: dump: 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:13.163 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:13.164 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:13.164 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:13.164 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:13.164 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:13.164 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:13.164 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:13.164 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.162+0000 7fe8c1ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:13.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.163+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fe8bc003f60 con 0x7fe8c414be90 2024-03-25T19:46:13.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.164+0000 7fe8c37fe640 10 monclient: get_auth_request con 0x7fe8b006dc70 auth_method 0 2024-03-25T19:46:13.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.164+0000 7fe8abfff640 10 client.4473.objecter ms_handle_connect 0x7fe8b006dc70 2024-03-25T19:46:13.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.171+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fe8bc054960 con 0x7fe8c414be90 2024-03-25T19:46:13.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.171+0000 7fe8abfff640 10 client.4473.objecter ms_dispatch 0x7fe8c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:13.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.171+0000 7fe8abfff640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:13.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.171+0000 7fe8abfff640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:13.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.171+0000 7fe8abfff640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T19:46:13.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.171+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8bc050470 con 0x7fe8c414be90 2024-03-25T19:46:13.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.171+0000 7fe8abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:13.172 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.171+0000 7fe8abfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:13.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.425+0000 7fe8c9f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:13.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.425+0000 7fe8c9f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:13.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.425+0000 7fe8c9f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:13.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.425+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8c40627b0 con 0x7fe8c414be90 2024-03-25T19:46:13.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.462+0000 7fe8abfff640 1 -- 172.21.15.69:0/4134135000 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fe8bc0523a0 con 0x7fe8c414be90 2024-03-25T19:46:13.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.462+0000 7fe8abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:13.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.462+0000 7fe8abfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:13.463 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:46:13.463 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:13.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.464+0000 7fe8c9f9d640 10 librados: watch_flush enter 2024-03-25T19:46:13.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.464+0000 7fe8c9f9d640 10 librados: watch_flush exit 2024-03-25T19:46:13.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.464+0000 7fe8c9f9d640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.464+0000 7fe8c9f9d640 10 client.4473.objecter successfully canceled tick 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.464+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 >> v1:172.21.15.169:6800/3072548259 conn(0x7fe8b006dc70 legacy=0x7fe8b0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.465+0000 7fe8c9f9d640 10 monclient: shutdown 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.465+0000 7fe8c9f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.465+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 >> v1:172.21.15.69:6789/0 conn(0x7fe8c414be90 legacy=0x7fe8c41e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.465+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 shutdown_connections 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.465+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 >> 172.21.15.69:0/4134135000 conn(0x7fe8c40617a0 msgr2=0x7fe8c4137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.465+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 shutdown_connections 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.465+0000 7fe8c9f9d640 1 -- 172.21.15.69:0/4134135000 wait complete. 2024-03-25T19:46:13.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:13.465+0000 7fe8c9f9d640 1 librados: shutdown 2024-03-25T19:46:13.476 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:20.282 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:20.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.393+0000 7f758c77b640 10 monclient: get_monmap_and_config 2024-03-25T19:46:20.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.393+0000 7f758c77b640 10 monclient: build_initial_monmap 2024-03-25T19:46:20.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.394+0000 7f758c77b640 10 monclient: monmap: 2024-03-25T19:46:20.394 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:20.394 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:20.394 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:46:20.394192+0000 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:46:20.394192+0000 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.394+0000 7f758c77b640 1 Processor -- start 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.394+0000 7f758c77b640 1 -- start start 2024-03-25T19:46:20.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.394+0000 7f758c77b640 10 monclient: init 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 10 monclient(hunting): picked mon.noname-a con 0x7f75840a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 10 monclient(hunting): picked mon.noname-c con 0x7f758413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 10 monclient(hunting): picked mon.noname-b con 0x7f75841430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7584142520 con 0x7f75840a98f0 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840b28a0 con 0x7f758413f710 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840b2a30 con 0x7f75841430c0 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:20.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.395+0000 7f758c77b640 10 monclient(hunting): authenticate will time out at 5192.663219s 2024-03-25T19:46:20.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.405+0000 7f7589d16640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f758413f710 0x7f75841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37118/0 (socket says 172.21.15.69:37118) 2024-03-25T19:46:20.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.405+0000 7f7589d16640 1 -- 172.21.15.69:0/922650521 learned_addr learned my addr 172.21.15.69:0/922650521 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:20.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.415+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403147319 0 0) 0x7f75840b28a0 con 0x7f758413f710 2024-03-25T19:46:20.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.415+0000 7f7589515640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.415+0000 7f7589515640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.415+0000 7f7589515640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.415+0000 7f7589515640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:46:20.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.415+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7570003520 con 0x7f758413f710 2024-03-25T19:46:20.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.421+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871451238 0 0) 0x7f7584142520 con 0x7f75840a98f0 2024-03-25T19:46:20.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.421+0000 7f7589515640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.421+0000 7f7589515640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.421+0000 7f7589515640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.421+0000 7f7589515640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.421+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75840b28a0 con 0x7f75840a98f0 2024-03-25T19:46:20.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.423+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216112505 0 0) 0x7f75840b2a30 con 0x7f75841430c0 2024-03-25T19:46:20.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.423+0000 7f7589515640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.423+0000 7f7589515640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.423+0000 7f7589515640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.423+0000 7f7589515640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:46:20.422 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.423+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584142520 con 0x7f75841430c0 2024-03-25T19:46:20.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593856883 0 0) 0x7f7570003520 con 0x7f758413f710 2024-03-25T19:46:20.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 >> v1:172.21.15.69:6790/0 conn(0x7f75841430c0 legacy=0x7f758414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 >> v1:172.21.15.69:6789/0 conn(0x7f75840a98f0 legacy=0x7f75840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 10 monclient: _finish_hunting 0 2024-03-25T19:46:20.423 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 1 monclient: found mon.noname-c 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840b44c0 con 0x7f758413f710 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 10 monclient: _finish_auth 0 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 10 monclient: _check_auth_tickets 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7574001690 con 0x7f758413f710 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f7589515640 10 monclient: dump: 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:20.424 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:20.425 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:20.425 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:20.425 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:20.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f758c77b640 5 monclient: authenticate success, global_id 4447 2024-03-25T19:46:20.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.424+0000 7f758c77b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75740017f0 con 0x7f758413f710 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f7589515640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f7589515640 1 -- 172.21.15.69:0/922650521 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7574001c00 con 0x7f758413f710 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f758c77b640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f758c77b640 4 set_mon_vals no callback set 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f7589515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f7589515640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f7589515640 10 monclient: dump: 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.441 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:20.442 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:20.442 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:20.442 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:20.442 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:20.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f758c77b640 10 monclient: shutdown 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f758c77b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.442+0000 7f758c77b640 1 -- 172.21.15.69:0/922650521 >> v1:172.21.15.169:6789/0 conn(0x7f758413f710 legacy=0x7f75841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.443+0000 7f758c77b640 1 -- 172.21.15.69:0/922650521 shutdown_connections 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.443+0000 7f758c77b640 1 -- 172.21.15.69:0/922650521 >> 172.21.15.69:0/922650521 conn(0x7f75840b7ea0 msgr2=0x7f75840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.443+0000 7f758c77b640 1 -- 172.21.15.69:0/922650521 shutdown_connections 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.443+0000 7f758c77b640 1 -- 172.21.15.69:0/922650521 wait complete. 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 10 monclient: build_initial_monmap 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 10 monclient: monmap: 2024-03-25T19:46:20.446 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 1 librados: starting msgr at 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 1 librados: starting objecter 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 1 Processor -- start 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 1 -- start start 2024-03-25T19:46:20.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 1 librados: setting wanted keys 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 1 librados: calling monclient init 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.444+0000 7f758c77b640 10 monclient: init 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 10 monclient(hunting): picked mon.noname-c con 0x7f75840a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 10 monclient(hunting): picked mon.noname-b con 0x7f758413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 10 monclient(hunting): picked mon.noname-a con 0x7f75841430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7584139070 con 0x7f75841430c0 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7584133710 con 0x7f758413f710 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75841338a0 con 0x7f75840a98f0 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:20.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.445+0000 7f758c77b640 10 monclient(hunting): authenticate will time out at 5192.712998s 2024-03-25T19:46:20.456 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.457+0000 7f7589d16640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f758413f710 0x7f75841daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37170/0 (socket says 172.21.15.69:37170) 2024-03-25T19:46:20.456 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.457+0000 7f7589d16640 1 -- 172.21.15.69:0/1465463523 learned_addr learned my addr 172.21.15.69:0/1465463523 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:20.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.460+0000 7f757affd640 10 client.?.objecter ms_handle_connect 0x7f758413f710 2024-03-25T19:46:20.460 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.461+0000 7f757affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:20.469 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.470+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305659048 0 0) 0x7f7584133710 con 0x7f758413f710 2024-03-25T19:46:20.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.470+0000 7f757affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.470+0000 7f757affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.470+0000 7f757affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.470+0000 7f757affd640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:46:20.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.470+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7568004eb0 con 0x7f758413f710 2024-03-25T19:46:20.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.470+0000 7f757affd640 10 client.?.objecter ms_handle_connect 0x7f75840a98f0 2024-03-25T19:46:20.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.470+0000 7f757affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:20.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.475+0000 7f757affd640 10 client.?.objecter ms_handle_connect 0x7f75841430c0 2024-03-25T19:46:20.474 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.475+0000 7f757affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:20.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.486+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648968738 0 0) 0x7f75841338a0 con 0x7f75840a98f0 2024-03-25T19:46:20.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.486+0000 7f757affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.486+0000 7f757affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.486+0000 7f757affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.486+0000 7f757affd640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:46:20.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.486+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584133710 con 0x7f75840a98f0 2024-03-25T19:46:20.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.487+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954360008 0 0) 0x7f7584139070 con 0x7f75841430c0 2024-03-25T19:46:20.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.487+0000 7f757affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.487+0000 7f757affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.487+0000 7f757affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.487+0000 7f757affd640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:46:20.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.487+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75841338a0 con 0x7f75841430c0 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184088407 0 0) 0x7f7568004eb0 con 0x7f758413f710 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7584139070 con 0x7f758413f710 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7574001bc0 con 0x7f758413f710 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 10 monclient(hunting): dump: 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:20.507 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3225622282 0 0) 0x7f75841338a0 con 0x7f75841430c0 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7568004eb0 con 0x7f75841430c0 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f757c001bd0 con 0x7f75841430c0 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.507+0000 7f757affd640 10 monclient(hunting): dump: 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.508 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.509 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:20.509 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:20.509 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:20.509 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:20.509 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:20.509 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.509+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596284760 0 0) 0x7f7584133710 con 0x7f75840a98f0 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.509+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75841338a0 con 0x7f75840a98f0 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.509+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7580001970 con 0x7f75840a98f0 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.509+0000 7f757affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.509+0000 7f757affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.509+0000 7f757affd640 10 monclient(hunting): dump: 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:20.510 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:20.511 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:20.511 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:20.514 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1619731999 0 0) 0x7f7584139070 con 0x7f758413f710 2024-03-25T19:46:20.514 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 >> v1:172.21.15.69:6790/0 conn(0x7f75840a98f0 legacy=0x7f75841378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.514 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 >> v1:172.21.15.69:6789/0 conn(0x7f75841430c0 legacy=0x7f7584133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.514 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 1 monclient: found mon.b 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7584133a30 con 0x7f758413f710 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 10 monclient: _finish_auth 0 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f757affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 10 monclient: _renew_subs 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75840b7100 con 0x7f758413f710 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 10 monclient: _renew_subs 2024-03-25T19:46:20.515 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:20.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7584136e60 con 0x7f758413f710 2024-03-25T19:46:20.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.513+0000 7f758c77b640 1 librados: init done 2024-03-25T19:46:20.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.514+0000 7f758c77b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:20.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.514+0000 7f758c77b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:20.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.514+0000 7f758c77b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:20.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.514+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7584074cc0 con 0x7f758413f710 2024-03-25T19:46:20.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.809+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75740011f0 con 0x7f758413f710 2024-03-25T19:46:20.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.809+0000 7f757affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:20.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.809+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f7574004c30 con 0x7f758413f710 2024-03-25T19:46:20.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.809+0000 7f757affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.809+0000 7f757affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:20.808 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.809+0000 7f757affd640 10 monclient: dump: 2024-03-25T19:46:20.808 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:20.809 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.809+0000 7f7588d14640 4 set_mon_vals no callback set 2024-03-25T19:46:20.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.810+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f757401ac40 con 0x7f758413f710 2024-03-25T19:46:20.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.810+0000 7f758a517640 10 monclient: get_auth_request con 0x7f756806e060 auth_method 0 2024-03-25T19:46:20.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.811+0000 7f757affd640 10 client.4450.objecter ms_handle_connect 0x7f756806e060 2024-03-25T19:46:20.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.814+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f7574054980 con 0x7f758413f710 2024-03-25T19:46:20.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.814+0000 7f757affd640 10 client.4450.objecter ms_dispatch 0x7f7584000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:20.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.814+0000 7f757affd640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:20.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.814+0000 7f757affd640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:20.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.814+0000 7f757affd640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T19:46:20.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.815+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75740502c0 con 0x7f758413f710 2024-03-25T19:46:20.814 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.815+0000 7f757affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:20.814 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:20.815+0000 7f757affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:21.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.027+0000 7f758c77b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:21.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.027+0000 7f758c77b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:21.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.027+0000 7f758c77b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:21.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.027+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75840627b0 con 0x7f758413f710 2024-03-25T19:46:21.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.107+0000 7f757affd640 1 -- 172.21.15.69:0/1465463523 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f75740523c0 con 0x7f758413f710 2024-03-25T19:46:21.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.108+0000 7f757affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:21.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.108+0000 7f757affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:21.107 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:46:21.107 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:21.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.110+0000 7f758c77b640 10 librados: watch_flush enter 2024-03-25T19:46:21.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.110+0000 7f758c77b640 10 librados: watch_flush exit 2024-03-25T19:46:21.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.110+0000 7f758c77b640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T19:46:21.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.110+0000 7f758c77b640 10 client.4450.objecter successfully canceled tick 2024-03-25T19:46:21.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.110+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 >> v1:172.21.15.169:6800/3072548259 conn(0x7f756806e060 legacy=0x7f7568070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:21.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.110+0000 7f758c77b640 10 monclient: shutdown 2024-03-25T19:46:21.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.110+0000 7f758c77b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:21.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.110+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 >> v1:172.21.15.169:6789/0 conn(0x7f758413f710 legacy=0x7f75841daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:21.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.111+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 shutdown_connections 2024-03-25T19:46:21.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.111+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 >> 172.21.15.69:0/1465463523 conn(0x7f75840b7ea0 msgr2=0x7f7584074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:21.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.111+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 shutdown_connections 2024-03-25T19:46:21.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.111+0000 7f758c77b640 1 -- 172.21.15.69:0/1465463523 wait complete. 2024-03-25T19:46:21.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:21.111+0000 7f758c77b640 1 librados: shutdown 2024-03-25T19:46:21.121 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:27.876 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:05.840 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.929 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:27.930 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:05.894 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.970 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:05.934 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.979 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:05.933 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.985 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:05.949 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.990 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:05.944 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.991 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:05.955 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:28.012 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:05.976 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:28.018 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:05.972 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:28.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.034+0000 7fa39c523640 10 monclient: get_monmap_and_config 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.034+0000 7fa39c523640 10 monclient: build_initial_monmap 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.035+0000 7fa39c523640 10 monclient: monmap: 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:46:28.036186+0000 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:46:28.036186+0000 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:46:28.035 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:28.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.035+0000 7fa39c523640 1 Processor -- start 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.035+0000 7fa39c523640 1 -- start start 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.035+0000 7fa39c523640 10 monclient: init 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 10 monclient(hunting): picked mon.noname-c con 0x7fa3940b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 10 monclient(hunting): picked mon.noname-b con 0x7fa3940b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 10 monclient(hunting): picked mon.noname-a con 0x7fa39414be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3940af940 con 0x7fa39414be90 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3940b0a70 con 0x7fa3940b4d60 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3940b0d50 con 0x7fa3940b1730 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:28.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.036+0000 7fa39c523640 10 monclient(hunting): authenticate will time out at 5200.305292s 2024-03-25T19:46:28.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.061+0000 7fa39aac0640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa39414be90 0x7fa39414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53366/0 (socket says 172.21.15.69:53366) 2024-03-25T19:46:28.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.061+0000 7fa39aac0640 1 -- 172.21.15.69:0/1951235620 learned_addr learned my addr 172.21.15.69:0/1951235620 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:28.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.082+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606357596 0 0) 0x7fa3940b0a70 con 0x7fa3940b4d60 2024-03-25T19:46:28.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.082+0000 7fa3992bd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.082+0000 7fa3992bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.082+0000 7fa3992bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.082+0000 7fa3992bd640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:46:28.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.082+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa37c003520 con 0x7fa3940b4d60 2024-03-25T19:46:28.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.090+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569564576 0 0) 0x7fa3940af940 con 0x7fa39414be90 2024-03-25T19:46:28.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.090+0000 7fa3992bd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.090+0000 7fa3992bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.090+0000 7fa3992bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.090+0000 7fa3992bd640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:46:28.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.090+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3940b0a70 con 0x7fa39414be90 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333279479 0 0) 0x7fa37c003520 con 0x7fa3940b4d60 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 >> v1:172.21.15.69:6790/0 conn(0x7fa3940b1730 legacy=0x7fa3940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 >> v1:172.21.15.69:6789/0 conn(0x7fa39414be90 legacy=0x7fa39414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 1 monclient: found mon.noname-c 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3940b1030 con 0x7fa3940b4d60 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 10 monclient: _finish_auth 0 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fa38c001690 con 0x7fa3940b4d60 2024-03-25T19:46:28.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa3992bd640 10 monclient: dump: 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa39c523640 5 monclient: authenticate success, global_id 4453 2024-03-25T19:46:28.098 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.096+0000 7fa39c523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:28.102 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.102+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa38c0017f0 con 0x7fa3940b4d60 2024-03-25T19:46:28.102 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.102+0000 7fa3992bd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:28.102 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.102+0000 7fa3992bd640 1 -- 172.21.15.69:0/1951235620 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fa38c001c00 con 0x7fa3940b4d60 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.102+0000 7fa3992bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.102+0000 7fa3992bd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.102+0000 7fa3992bd640 10 monclient: dump: 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.102+0000 7fa39c523640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:28.103 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.102+0000 7fa39c523640 4 set_mon_vals no callback set 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.103+0000 7fa39c523640 10 monclient: shutdown 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.103+0000 7fa39c523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.103+0000 7fa39c523640 1 -- 172.21.15.69:0/1951235620 >> v1:172.21.15.169:6789/0 conn(0x7fa3940b4d60 legacy=0x7fa39413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.103+0000 7fa39c523640 1 -- 172.21.15.69:0/1951235620 shutdown_connections 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.103+0000 7fa39c523640 1 -- 172.21.15.69:0/1951235620 >> 172.21.15.69:0/1951235620 conn(0x7fa3940617a0 msgr2=0x7fa394061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.103+0000 7fa39c523640 1 -- 172.21.15.69:0/1951235620 shutdown_connections 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.103+0000 7fa39c523640 1 -- 172.21.15.69:0/1951235620 wait complete. 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 10 monclient: build_initial_monmap 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 10 monclient: monmap: 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:28.106 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 1 librados: starting msgr at 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 1 librados: starting objecter 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 1 Processor -- start 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 1 -- start start 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 1 librados: setting wanted keys 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 1 librados: calling monclient init 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.104+0000 7fa39c523640 10 monclient: init 2024-03-25T19:46:28.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 10 monclient(hunting): picked mon.noname-c con 0x7fa3940b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 10 monclient(hunting): picked mon.noname-b con 0x7fa3940b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 10 monclient(hunting): picked mon.noname-a con 0x7fa39414be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa394143350 con 0x7fa39414be90 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa394142110 con 0x7fa3940b4d60 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3941408f0 con 0x7fa3940b1730 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:28.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.105+0000 7fa39c523640 10 monclient(hunting): authenticate will time out at 5200.373919s 2024-03-25T19:46:28.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.125+0000 7fa39a2bf640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fa3940b1730 0x7fa394134160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:59632/0 (socket says 172.21.15.69:59632) 2024-03-25T19:46:28.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.125+0000 7fa39a2bf640 1 -- 172.21.15.69:0/3758205721 learned_addr learned my addr 172.21.15.69:0/3758205721 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:28.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.128+0000 7fa392ffd640 10 client.?.objecter ms_handle_connect 0x7fa3940b1730 2024-03-25T19:46:28.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.128+0000 7fa392ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:28.144 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.144+0000 7fa392ffd640 10 client.?.objecter ms_handle_connect 0x7fa3940b4d60 2024-03-25T19:46:28.144 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.144+0000 7fa392ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:28.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.153+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546102856 0 0) 0x7fa3941408f0 con 0x7fa3940b1730 2024-03-25T19:46:28.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.153+0000 7fa392ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.153+0000 7fa392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.153+0000 7fa392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.153+0000 7fa392ffd640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:46:28.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.153+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa380004eb0 con 0x7fa3940b1730 2024-03-25T19:46:28.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.158+0000 7fa392ffd640 10 client.?.objecter ms_handle_connect 0x7fa39414be90 2024-03-25T19:46:28.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.158+0000 7fa392ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:28.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.162+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 373733286 0 0) 0x7fa380004eb0 con 0x7fa3940b1730 2024-03-25T19:46:28.162 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.162+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3941408f0 con 0x7fa3940b1730 2024-03-25T19:46:28.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.166+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595751830 0 0) 0x7fa394142110 con 0x7fa3940b4d60 2024-03-25T19:46:28.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.166+0000 7fa392ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.166+0000 7fa392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.166+0000 7fa392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.166+0000 7fa392ffd640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:46:28.167 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.166+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa380004eb0 con 0x7fa3940b4d60 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.169+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fa38c001c80 con 0x7fa3940b1730 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.169+0000 7fa392ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.169+0000 7fa392ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.169+0000 7fa392ffd640 10 monclient(hunting): dump: 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:28.169 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:28.170 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.169+0000 7fa392ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:28.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.176+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730522709 0 0) 0x7fa394143350 con 0x7fa39414be90 2024-03-25T19:46:28.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.176+0000 7fa392ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.176+0000 7fa392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.176+0000 7fa392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.176+0000 7fa392ffd640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:46:28.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.176+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa394142110 con 0x7fa39414be90 2024-03-25T19:46:28.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2584300272 0 0) 0x7fa3941408f0 con 0x7fa3940b1730 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 >> v1:172.21.15.169:6789/0 conn(0x7fa3940b4d60 legacy=0x7fa394142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 >> v1:172.21.15.69:6789/0 conn(0x7fa39414be90 legacy=0x7fa394142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 1 monclient: found mon.c 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394140eb0 con 0x7fa3940b1730 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 5 monclient: authenticate success, global_id 4469 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 10 monclient: _renew_subs 2024-03-25T19:46:28.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:28.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa394141100 con 0x7fa3940b1730 2024-03-25T19:46:28.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:28.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 10 monclient: _renew_subs 2024-03-25T19:46:28.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:28.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa394134ab0 con 0x7fa3940b1730 2024-03-25T19:46:28.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.177+0000 7fa39c523640 1 librados: init done 2024-03-25T19:46:28.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.178+0000 7fa39c523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:28.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.178+0000 7fa39c523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:28.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.178+0000 7fa39c523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:28.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.178+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3941401f0 con 0x7fa3940b1730 2024-03-25T19:46:28.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.480+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa38c0011f0 con 0x7fa3940b1730 2024-03-25T19:46:28.480 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.480+0000 7fa392ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.480+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fa38c003ac0 con 0x7fa3940b1730 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.480+0000 7fa392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.480+0000 7fa392ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.480+0000 7fa392ffd640 10 monclient: dump: 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:28.481 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:28.482 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.480+0000 7fa398abc640 4 set_mon_vals no callback set 2024-03-25T19:46:28.482 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.481+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fa38c003ce0 con 0x7fa3940b1730 2024-03-25T19:46:28.482 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.482+0000 7fa399abe640 10 monclient: get_auth_request con 0x7fa38006dda0 auth_method 0 2024-03-25T19:46:28.482 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.482+0000 7fa392ffd640 10 client.4469.objecter ms_handle_connect 0x7fa38006dda0 2024-03-25T19:46:28.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.488+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fa38c054970 con 0x7fa3940b1730 2024-03-25T19:46:28.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.488+0000 7fa392ffd640 10 client.4469.objecter ms_dispatch 0x7fa394000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:28.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.488+0000 7fa392ffd640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:28.488 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.488+0000 7fa392ffd640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:28.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.488+0000 7fa392ffd640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-25T19:46:28.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.488+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa38c0502b0 con 0x7fa3940b1730 2024-03-25T19:46:28.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.488+0000 7fa392ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:28.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.488+0000 7fa392ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:28.749 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.749+0000 7fa39c523640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.749 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.749+0000 7fa39c523640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.749 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.749+0000 7fa39c523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:28.749 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.749+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa394140a80 con 0x7fa3940b1730 2024-03-25T19:46:28.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.783+0000 7fa392ffd640 1 -- 172.21.15.69:0/3758205721 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fa38c0523b0 con 0x7fa3940b1730 2024-03-25T19:46:28.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.783+0000 7fa392ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.783+0000 7fa392ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:28.783 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:46:28.784 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:28.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 10 librados: watch_flush enter 2024-03-25T19:46:28.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 10 librados: watch_flush exit 2024-03-25T19:46:28.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-25T19:46:28.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 10 client.4469.objecter successfully canceled tick 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 >> v1:172.21.15.169:6800/3072548259 conn(0x7fa38006dda0 legacy=0x7fa380070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 10 monclient: shutdown 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 >> v1:172.21.15.69:6790/0 conn(0x7fa3940b1730 legacy=0x7fa394134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 shutdown_connections 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 >> 172.21.15.69:0/3758205721 conn(0x7fa3940617a0 msgr2=0x7fa39413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.786+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 shutdown_connections 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.787+0000 7fa39c523640 1 -- 172.21.15.69:0/3758205721 wait complete. 2024-03-25T19:46:28.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:28.787+0000 7fa39c523640 1 librados: shutdown 2024-03-25T19:46:28.798 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:35.603 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.721+0000 7fc53094d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.721+0000 7fc53094d640 10 monclient: build_initial_monmap 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.721+0000 7fc53094d640 10 monclient: monmap: 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:46:35.722295+0000 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:46:35.722295+0000 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:35.721 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:35.722 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:46:35.722 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:46:35.722 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.721+0000 7fc53094d640 1 Processor -- start 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.721+0000 7fc53094d640 1 -- start start 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.721+0000 7fc53094d640 10 monclient: init 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 10 monclient(hunting): picked mon.noname-a con 0x7fc52c0a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 10 monclient(hunting): picked mon.noname-b con 0x7fc52c13f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 10 monclient(hunting): picked mon.noname-c con 0x7fc52c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c142520 con 0x7fc52c0a98f0 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c0b28a0 con 0x7fc52c1430c0 2024-03-25T19:46:35.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c0b2a30 con 0x7fc52c13f710 2024-03-25T19:46:35.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:35.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.722+0000 7fc53094d640 10 monclient(hunting): authenticate will time out at 5207.991265s 2024-03-25T19:46:35.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.726+0000 7fc52ad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc52c1430c0 0x7fc52c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:52756/0 (socket says 172.21.15.69:52756) 2024-03-25T19:46:35.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.726+0000 7fc52ad9d640 1 -- 172.21.15.69:0/3999062571 learned_addr learned my addr 172.21.15.69:0/3999062571 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:35.741 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.741+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363828951 0 0) 0x7fc52c0b28a0 con 0x7fc52c1430c0 2024-03-25T19:46:35.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.741+0000 7fc529d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.741+0000 7fc529d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.741+0000 7fc529d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.741+0000 7fc529d9b640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:46:35.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.741+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc518003520 con 0x7fc52c1430c0 2024-03-25T19:46:35.750 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.750+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039418776 0 0) 0x7fc52c142520 con 0x7fc52c0a98f0 2024-03-25T19:46:35.750 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.750+0000 7fc529d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.750+0000 7fc529d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.750+0000 7fc529d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.750+0000 7fc529d9b640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:46:35.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.750+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52c0b28a0 con 0x7fc52c0a98f0 2024-03-25T19:46:35.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.751+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877213152 0 0) 0x7fc52c0b2a30 con 0x7fc52c13f710 2024-03-25T19:46:35.751 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.751+0000 7fc529d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.751+0000 7fc529d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.751+0000 7fc529d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.751+0000 7fc529d9b640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:46:35.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.751+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52c142520 con 0x7fc52c13f710 2024-03-25T19:46:35.764 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944271387 0 0) 0x7fc518003520 con 0x7fc52c1430c0 2024-03-25T19:46:35.764 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 >> v1:172.21.15.69:6790/0 conn(0x7fc52c13f710 legacy=0x7fc52c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.764 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 >> v1:172.21.15.69:6789/0 conn(0x7fc52c0a98f0 legacy=0x7fc52c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 1 monclient: found mon.noname-c 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c0b44c0 con 0x7fc52c1430c0 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 10 monclient: _finish_auth 0 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc524001a10 con 0x7fc52c1430c0 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc529d9b640 10 monclient: dump: 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:35.765 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:35.766 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.766 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:35.766 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:35.766 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:35.766 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:35.766 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:35.766 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc53094d640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:46:35.766 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.764+0000 7fc53094d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc524001b70 con 0x7fc52c1430c0 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc529d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc529d9b640 1 -- 172.21.15.69:0/3999062571 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc524001f80 con 0x7fc52c1430c0 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc53094d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc53094d640 4 set_mon_vals no callback set 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc529d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc529d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc529d9b640 10 monclient: dump: 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:35.773 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:35.774 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.774 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:35.774 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:35.774 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:35.774 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:35.774 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:35.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc53094d640 10 monclient: shutdown 2024-03-25T19:46:35.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc53094d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:35.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.773+0000 7fc53094d640 1 -- 172.21.15.69:0/3999062571 >> v1:172.21.15.169:6789/0 conn(0x7fc52c1430c0 legacy=0x7fc52c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.774+0000 7fc53094d640 1 -- 172.21.15.69:0/3999062571 shutdown_connections 2024-03-25T19:46:35.775 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.774+0000 7fc53094d640 1 -- 172.21.15.69:0/3999062571 >> 172.21.15.69:0/3999062571 conn(0x7fc52c0b7ea0 msgr2=0x7fc52c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.774+0000 7fc53094d640 1 -- 172.21.15.69:0/3999062571 shutdown_connections 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.774+0000 7fc53094d640 1 -- 172.21.15.69:0/3999062571 wait complete. 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 10 monclient: build_initial_monmap 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 10 monclient: monmap: 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 1 librados: starting msgr at 2024-03-25T19:46:35.776 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 1 librados: starting objecter 2024-03-25T19:46:35.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 1 Processor -- start 2024-03-25T19:46:35.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 1 -- start start 2024-03-25T19:46:35.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 1 librados: setting wanted keys 2024-03-25T19:46:35.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 1 librados: calling monclient init 2024-03-25T19:46:35.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.775+0000 7fc53094d640 10 monclient: init 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 10 monclient(hunting): picked mon.noname-c con 0x7fc52c0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 10 monclient(hunting): picked mon.noname-b con 0x7fc52c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 10 monclient(hunting): picked mon.noname-a con 0x7fc52c1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c1d7500 con 0x7fc52c1430c0 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c1394a0 con 0x7fc52c13f710 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c138b50 con 0x7fc52c0a98f0 2024-03-25T19:46:35.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:35.779 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.776+0000 7fc53094d640 10 monclient(hunting): authenticate will time out at 5208.045294s 2024-03-25T19:46:35.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.783+0000 7fc52a59c640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fc52c0a98f0 0x7fc52c138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:59850/0 (socket says 172.21.15.69:59850) 2024-03-25T19:46:35.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.783+0000 7fc52a59c640 1 -- 172.21.15.69:0/2992970217 learned_addr learned my addr 172.21.15.69:0/2992970217 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:35.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.784+0000 7fc5237fe640 10 client.?.objecter ms_handle_connect 0x7fc52c13f710 2024-03-25T19:46:35.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.784+0000 7fc5237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.785 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.785+0000 7fc5237fe640 10 client.?.objecter ms_handle_connect 0x7fc52c0a98f0 2024-03-25T19:46:35.785 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.785+0000 7fc5237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.792+0000 7fc5237fe640 10 client.?.objecter ms_handle_connect 0x7fc52c1430c0 2024-03-25T19:46:35.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.792+0000 7fc5237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.796+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625407564 0 0) 0x7fc52c138b50 con 0x7fc52c0a98f0 2024-03-25T19:46:35.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.796+0000 7fc5237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.796+0000 7fc5237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.796+0000 7fc5237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.796+0000 7fc5237fe640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:46:35.796 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.796+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50c004eb0 con 0x7fc52c0a98f0 2024-03-25T19:46:35.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.799+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729502292 0 0) 0x7fc52c1394a0 con 0x7fc52c13f710 2024-03-25T19:46:35.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.799+0000 7fc5237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.799+0000 7fc5237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.799+0000 7fc5237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.799+0000 7fc5237fe640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:46:35.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.799+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52c138b50 con 0x7fc52c13f710 2024-03-25T19:46:35.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.811+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141145888 0 0) 0x7fc52c1d7500 con 0x7fc52c1430c0 2024-03-25T19:46:35.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.811+0000 7fc5237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.811+0000 7fc5237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.811+0000 7fc5237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.811+0000 7fc5237fe640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:46:35.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.811+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52c1394a0 con 0x7fc52c1430c0 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.819+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3789102107 0 0) 0x7fc52c138b50 con 0x7fc52c13f710 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.819+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc52c1d7500 con 0x7fc52c13f710 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.819+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc514001870 con 0x7fc52c13f710 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.819+0000 7fc5237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.819+0000 7fc5237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.819+0000 7fc5237fe640 10 monclient(hunting): dump: 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:35.819 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:35.820 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:35.820 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.820 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:35.820 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:35.820 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:35.820 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:35.820 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:35.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.819+0000 7fc5237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:35.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.825+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339896716 0 0) 0x7fc50c004eb0 con 0x7fc52c0a98f0 2024-03-25T19:46:35.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.825+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc52c138b50 con 0x7fc52c0a98f0 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.825+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc51c001a30 con 0x7fc52c0a98f0 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.825+0000 7fc5237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.825+0000 7fc5237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.825+0000 7fc5237fe640 10 monclient(hunting): dump: 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:35.826 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4065367969 0 0) 0x7fc52c1d7500 con 0x7fc52c13f710 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 >> v1:172.21.15.69:6790/0 conn(0x7fc52c0a98f0 legacy=0x7fc52c138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 >> v1:172.21.15.69:6789/0 conn(0x7fc52c1430c0 legacy=0x7fc52c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 1 monclient: found mon.b 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c1d8380 con 0x7fc52c13f710 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc5237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 5 monclient: authenticate success, global_id 4462 2024-03-25T19:46:35.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 10 monclient: _renew_subs 2024-03-25T19:46:35.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:35.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52c139710 con 0x7fc52c13f710 2024-03-25T19:46:35.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:35.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 10 monclient: _renew_subs 2024-03-25T19:46:35.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:35.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52c1d7750 con 0x7fc52c13f710 2024-03-25T19:46:35.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.833+0000 7fc53094d640 1 librados: init done 2024-03-25T19:46:35.835 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.834+0000 7fc53094d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:35.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.834+0000 7fc53094d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:35.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.834+0000 7fc53094d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:35.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:35.834+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52c0627b0 con 0x7fc52c13f710 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.133+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc514003610 con 0x7fc52c13f710 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.133+0000 7fc5237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.133+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc514001d80 con 0x7fc52c13f710 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.133+0000 7fc5237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.133+0000 7fc5237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.133+0000 7fc5237fe640 10 monclient: dump: 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:36.133 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:36.134 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:36.134 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:36.134 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:36.134 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.133+0000 7fc52959a640 4 set_mon_vals no callback set 2024-03-25T19:46:36.134 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.134+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fc514003b20 con 0x7fc52c13f710 2024-03-25T19:46:36.134 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.134+0000 7fc52a59c640 10 monclient: get_auth_request con 0x7fc50c06df70 auth_method 0 2024-03-25T19:46:36.135 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.135+0000 7fc5237fe640 10 client.4462.objecter ms_handle_connect 0x7fc50c06df70 2024-03-25T19:46:36.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.139+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fc514053530 con 0x7fc52c13f710 2024-03-25T19:46:36.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.139+0000 7fc5237fe640 10 client.4462.objecter ms_dispatch 0x7fc52c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:36.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.139+0000 7fc5237fe640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:36.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.139+0000 7fc5237fe640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:36.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.139+0000 7fc5237fe640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T19:46:36.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.140+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc51404f040 con 0x7fc52c13f710 2024-03-25T19:46:36.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.140+0000 7fc5237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:36.140 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.140+0000 7fc5237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:36.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.394+0000 7fc53094d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:36.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.394+0000 7fc53094d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:36.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.394+0000 7fc53094d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:46:36.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.394+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc52c069f40 con 0x7fc52c13f710 2024-03-25T19:46:36.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.425+0000 7fc5237fe640 1 -- 172.21.15.69:0/2992970217 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fc514050f70 con 0x7fc52c13f710 2024-03-25T19:46:36.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.425+0000 7fc5237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:36.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.425+0000 7fc5237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:36.426 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:46:36.426 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 10 librados: watch_flush enter 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 10 librados: watch_flush exit 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 10 client.4462.objecter successfully canceled tick 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 >> v1:172.21.15.169:6800/3072548259 conn(0x7fc50c06df70 legacy=0x7fc50c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 10 monclient: shutdown 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 >> v1:172.21.15.169:6789/0 conn(0x7fc52c13f710 legacy=0x7fc52c1d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 shutdown_connections 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 >> 172.21.15.69:0/2992970217 conn(0x7fc52c0b7ea0 msgr2=0x7fc52c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:36.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 shutdown_connections 2024-03-25T19:46:36.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 1 -- 172.21.15.69:0/2992970217 wait complete. 2024-03-25T19:46:36.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:36.428+0000 7fc53094d640 1 librados: shutdown 2024-03-25T19:46:36.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:43.246 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.361+0000 7f992fb7b640 10 monclient: get_monmap_and_config 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.361+0000 7f992fb7b640 10 monclient: build_initial_monmap 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.361+0000 7f992fb7b640 10 monclient: monmap: 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:46:43.362343+0000 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:46:43.362343+0000 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:43.361 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:46:43.362 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:46:43.362 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:43.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.361+0000 7f992fb7b640 1 Processor -- start 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 1 -- start start 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 10 monclient: init 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 10 monclient(hunting): picked mon.noname-c con 0x7f99280b1ef0 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 10 monclient(hunting): picked mon.noname-a con 0x7f99281473a0 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 10 monclient(hunting): picked mon.noname-b con 0x7f9928150070 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99280b2570 con 0x7f99281473a0 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9928146c20 con 0x7f99280b1ef0 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9928146e80 con 0x7f9928150070 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:43.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.362+0000 7f992fb7b640 10 monclient(hunting): authenticate will time out at 5215.631597s 2024-03-25T19:46:43.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.365+0000 7f992d917640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f99280b1ef0 0x7f99280b28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:55048/0 (socket says 172.21.15.69:55048) 2024-03-25T19:46:43.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.366+0000 7f992d917640 1 -- 172.21.15.69:0/913481677 learned_addr learned my addr 172.21.15.69:0/913481677 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:43.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.374+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256686664 0 0) 0x7f9928146c20 con 0x7f99280b1ef0 2024-03-25T19:46:43.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.374+0000 7f992c915640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.374+0000 7f992c915640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.374+0000 7f992c915640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.374+0000 7f992c915640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:46:43.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.374+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9914003520 con 0x7f99280b1ef0 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1755313366 0 0) 0x7f9914003520 con 0x7f99280b1ef0 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 >> v1:172.21.15.69:6790/0 conn(0x7f9928150070 legacy=0x7f9928152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 >> v1:172.21.15.69:6789/0 conn(0x7f99281473a0 legacy=0x7f9928143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 10 monclient: _finish_hunting 0 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 1 monclient: found mon.noname-c 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992806e1b0 con 0x7f99280b1ef0 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 10 monclient: _finish_auth 0 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 10 monclient: _check_auth_tickets 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.382+0000 7f992c915640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:43.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.383+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9924001690 con 0x7f99280b1ef0 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.383+0000 7f992c915640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.383+0000 7f992c915640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.383+0000 7f992c915640 10 monclient: dump: 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.383+0000 7f992fb7b640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:46:43.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.383+0000 7f992fb7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:43.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.388+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9924001860 con 0x7f99280b1ef0 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.388+0000 7f992c915640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.388+0000 7f992c915640 1 -- 172.21.15.69:0/913481677 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9924001cb0 con 0x7f99280b1ef0 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.388+0000 7f992c915640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.388+0000 7f992c915640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.388+0000 7f992c915640 10 monclient: dump: 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:43.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.388+0000 7f992fb7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:43.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.388+0000 7f992fb7b640 4 set_mon_vals no callback set 2024-03-25T19:46:43.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.389+0000 7f992fb7b640 10 monclient: shutdown 2024-03-25T19:46:43.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.389+0000 7f992fb7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:43.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.389+0000 7f992fb7b640 1 -- 172.21.15.69:0/913481677 >> v1:172.21.15.169:6789/0 conn(0x7f99280b1ef0 legacy=0x7f99280b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.389+0000 7f992fb7b640 1 -- 172.21.15.69:0/913481677 shutdown_connections 2024-03-25T19:46:43.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.389+0000 7f992fb7b640 1 -- 172.21.15.69:0/913481677 >> 172.21.15.69:0/913481677 conn(0x7f99280b7ea0 msgr2=0x7f99280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:43.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.389+0000 7f992fb7b640 1 -- 172.21.15.69:0/913481677 shutdown_connections 2024-03-25T19:46:43.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.389+0000 7f992fb7b640 1 -- 172.21.15.69:0/913481677 wait complete. 2024-03-25T19:46:43.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.390+0000 7f992fb7b640 10 monclient: build_initial_monmap 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.390+0000 7f992fb7b640 10 monclient: monmap: 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 1 librados: starting msgr at 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 1 librados: starting objecter 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 1 Processor -- start 2024-03-25T19:46:43.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 1 -- start start 2024-03-25T19:46:43.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 1 librados: setting wanted keys 2024-03-25T19:46:43.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 1 librados: calling monclient init 2024-03-25T19:46:43.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 10 monclient: init 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.391+0000 7f992fb7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.392+0000 7f992fb7b640 10 monclient(hunting): picked mon.noname-b con 0x7f99280b1ef0 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.392+0000 7f992fb7b640 10 monclient(hunting): picked mon.noname-c con 0x7f99281473a0 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.392+0000 7f992fb7b640 10 monclient(hunting): picked mon.noname-a con 0x7f9928150070 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.392+0000 7f992fb7b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9928140a90 con 0x7f9928150070 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.392+0000 7f992fb7b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992813e430 con 0x7f99280b1ef0 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.392+0000 7f992fb7b640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992813f850 con 0x7f99281473a0 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.392+0000 7f992fb7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:43.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.392+0000 7f992fb7b640 10 monclient(hunting): authenticate will time out at 5215.660697s 2024-03-25T19:46:43.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.397+0000 7f992d116640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f99281473a0 0x7f99281df590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:47614/0 (socket says 172.21.15.69:47614) 2024-03-25T19:46:43.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.397+0000 7f992d116640 1 -- 172.21.15.69:0/623564319 learned_addr learned my addr 172.21.15.69:0/623564319 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:43.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.397+0000 7f991e7fc640 10 client.?.objecter ms_handle_connect 0x7f99281473a0 2024-03-25T19:46:43.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.397+0000 7f991e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:43.398 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.398+0000 7f991e7fc640 10 client.?.objecter ms_handle_connect 0x7f99280b1ef0 2024-03-25T19:46:43.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.398+0000 7f991e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:43.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.403+0000 7f991e7fc640 10 client.?.objecter ms_handle_connect 0x7f9928150070 2024-03-25T19:46:43.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.403+0000 7f991e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:43.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.410+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876976872 0 0) 0x7f992813e430 con 0x7f99280b1ef0 2024-03-25T19:46:43.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.410+0000 7f991e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.410+0000 7f991e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.410+0000 7f991e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.410+0000 7f991e7fc640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:46:43.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.410+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990c004eb0 con 0x7f99280b1ef0 2024-03-25T19:46:43.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.412+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507289139 0 0) 0x7f992813f850 con 0x7f99281473a0 2024-03-25T19:46:43.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.412+0000 7f991e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.412+0000 7f991e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.412+0000 7f991e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.412+0000 7f991e7fc640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:46:43.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.412+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f992813e430 con 0x7f99281473a0 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.423+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 979378439 0 0) 0x7f990c004eb0 con 0x7f99280b1ef0 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.423+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f992813f850 con 0x7f99280b1ef0 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.423+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9924001b70 con 0x7f99280b1ef0 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.423+0000 7f991e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.423+0000 7f991e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.423+0000 7f991e7fc640 10 monclient(hunting): dump: 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:43.424 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.423+0000 7f991e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.424+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899380548 0 0) 0x7f9928140a90 con 0x7f9928150070 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.424+0000 7f991e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.424+0000 7f991e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.424+0000 7f991e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.424+0000 7f991e7fc640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:46:43.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.424+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990c004eb0 con 0x7f9928150070 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.430+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3953695372 0 0) 0x7f992813e430 con 0x7f99281473a0 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.430+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9928140a90 con 0x7f99281473a0 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.430+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9918001980 con 0x7f99281473a0 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.430+0000 7f991e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.430+0000 7f991e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.430+0000 7f991e7fc640 10 monclient(hunting): dump: 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:43.431 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:43.442 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1396740323 0 0) 0x7f9928140a90 con 0x7f99281473a0 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 >> v1:172.21.15.169:6789/0 conn(0x7f99280b1ef0 legacy=0x7f99281dee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 >> v1:172.21.15.69:6789/0 conn(0x7f9928150070 legacy=0x7f99281e61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 1 monclient: found mon.c 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992813f9e0 con 0x7f99281473a0 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f991e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 5 monclient: authenticate success, global_id 4478 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 10 monclient: _renew_subs 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9928140eb0 con 0x7f99281473a0 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:43.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 10 monclient: _renew_subs 2024-03-25T19:46:43.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:43.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f992813eff0 con 0x7f99281473a0 2024-03-25T19:46:43.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.442+0000 7f992fb7b640 1 librados: init done 2024-03-25T19:46:43.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.443+0000 7f992fb7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:43.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.443+0000 7f992fb7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:43.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.443+0000 7f992fb7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:43.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.443+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9928141200 con 0x7f99281473a0 2024-03-25T19:46:43.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.746+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9918003610 con 0x7f99281473a0 2024-03-25T19:46:43.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.746+0000 7f991e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:43.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.746+0000 7f991ffff640 4 set_mon_vals no callback set 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.746+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9918003960 con 0x7f99281473a0 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.746+0000 7f991e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.746+0000 7f991e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.746+0000 7f991e7fc640 10 monclient: dump: 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:43.747 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:43.748 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.747+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f9918003b80 con 0x7f99281473a0 2024-03-25T19:46:43.748 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.747+0000 7f992d917640 10 monclient: get_auth_request con 0x7f990c06df00 auth_method 0 2024-03-25T19:46:43.748 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.748+0000 7f991e7fc640 10 client.4478.objecter ms_handle_connect 0x7f990c06df00 2024-03-25T19:46:43.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.753+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f9918054830 con 0x7f99281473a0 2024-03-25T19:46:43.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.753+0000 7f991e7fc640 10 client.4478.objecter ms_dispatch 0x7f9928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:43.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.753+0000 7f991e7fc640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:43.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.753+0000 7f991e7fc640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:43.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.753+0000 7f991e7fc640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T19:46:43.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.753+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9918050340 con 0x7f99281473a0 2024-03-25T19:46:43.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.753+0000 7f991e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:43.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:43.753+0000 7f991e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:44.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.005+0000 7f992fb7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:44.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.005+0000 7f992fb7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:44.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.005+0000 7f992fb7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:44.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.005+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f992813e9c0 con 0x7f99281473a0 2024-03-25T19:46:44.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.028+0000 7f991e7fc640 1 -- 172.21.15.69:0/623564319 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f9918052270 con 0x7f99281473a0 2024-03-25T19:46:44.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.028+0000 7f991e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:44.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.028+0000 7f991e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:44.029 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:46:44.029 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:44.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 10 librados: watch_flush enter 2024-03-25T19:46:44.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 10 librados: watch_flush exit 2024-03-25T19:46:44.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T19:46:44.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 10 client.4478.objecter successfully canceled tick 2024-03-25T19:46:44.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 >> v1:172.21.15.169:6800/3072548259 conn(0x7f990c06df00 legacy=0x7f990c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:44.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 10 monclient: shutdown 2024-03-25T19:46:44.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:44.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 >> v1:172.21.15.69:6790/0 conn(0x7f99281473a0 legacy=0x7f99281df590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:44.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 shutdown_connections 2024-03-25T19:46:44.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 >> 172.21.15.69:0/623564319 conn(0x7f99280b7ea0 msgr2=0x7f992813a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:44.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 shutdown_connections 2024-03-25T19:46:44.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 1 -- 172.21.15.69:0/623564319 wait complete. 2024-03-25T19:46:44.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:44.031+0000 7f992fb7b640 1 librados: shutdown 2024-03-25T19:46:44.043 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:48.000 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:25.964 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.008 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:25.972 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.029 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:25.993 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.055 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:26.009 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.059 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:26.014 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.068 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:26.032 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.072 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:26.036 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.092 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:26.046 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.101 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:26.065 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:50.850 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:50.970 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.970+0000 7f5b4a6cf640 10 monclient: get_monmap_and_config 2024-03-25T19:46:50.970 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.970+0000 7f5b4a6cf640 10 monclient: build_initial_monmap 2024-03-25T19:46:50.970 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.970+0000 7f5b4a6cf640 10 monclient: monmap: 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:46:50.971905+0000 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:46:50.971905+0000 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:46:50.971 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:50.972 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.971+0000 7f5b4a6cf640 1 Processor -- start 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.971+0000 7f5b4a6cf640 1 -- start start 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.971+0000 7f5b4a6cf640 10 monclient: init 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f5b440b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f5b440b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f5b4414be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b440af940 con 0x7f5b4414be90 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b440b0a70 con 0x7f5b440b1730 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b440b0d50 con 0x7f5b440b4d60 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:50.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.972+0000 7f5b4a6cf640 10 monclient(hunting): authenticate will time out at 5223.241269s 2024-03-25T19:46:50.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.978+0000 7f5b43fff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f5b440b4d60 0x7f5b4413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:50128/0 (socket says 172.21.15.69:50128) 2024-03-25T19:46:50.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.978+0000 7f5b43fff640 1 -- 172.21.15.69:0/2746801105 learned_addr learned my addr 172.21.15.69:0/2746801105 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:50.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.985+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641312699 0 0) 0x7f5b440b0d50 con 0x7f5b440b4d60 2024-03-25T19:46:50.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.985+0000 7f5b42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.985+0000 7f5b42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.985+0000 7f5b42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.985+0000 7f5b42ffd640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:46:50.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:50.985+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b2c003520 con 0x7f5b440b4d60 2024-03-25T19:46:51.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.000+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392460913 0 0) 0x7f5b440af940 con 0x7f5b4414be90 2024-03-25T19:46:51.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.000+0000 7f5b42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.000+0000 7f5b42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.000+0000 7f5b42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.000+0000 7f5b42ffd640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:46:51.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.000+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b440b0d50 con 0x7f5b4414be90 2024-03-25T19:46:51.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.007+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266101994 0 0) 0x7f5b440b0a70 con 0x7f5b440b1730 2024-03-25T19:46:51.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.007+0000 7f5b42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.007+0000 7f5b42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.007+0000 7f5b42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.007+0000 7f5b42ffd640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:46:51.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.007+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b440af940 con 0x7f5b440b1730 2024-03-25T19:46:51.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2993128811 0 0) 0x7f5b2c003520 con 0x7f5b440b4d60 2024-03-25T19:46:51.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 >> v1:172.21.15.169:6789/0 conn(0x7f5b440b1730 legacy=0x7f5b440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 >> v1:172.21.15.69:6789/0 conn(0x7f5b4414be90 legacy=0x7f5b4414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:51.009 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 1 monclient: found mon.noname-b 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b440b1030 con 0x7f5b440b4d60 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5b34001690 con 0x7f5b440b4d60 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b42ffd640 10 monclient: dump: 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:51.010 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:51.011 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:51.011 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:51.011 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:51.011 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:51.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b4a6cf640 5 monclient: authenticate success, global_id 4481 2024-03-25T19:46:51.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.009+0000 7f5b4a6cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:51.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.017+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b340017f0 con 0x7f5b440b4d60 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.017+0000 7f5b42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.017+0000 7f5b42ffd640 1 -- 172.21.15.69:0/2746801105 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5b34001c00 con 0x7f5b440b4d60 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.017+0000 7f5b42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.017+0000 7f5b42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.017+0000 7f5b42ffd640 10 monclient: dump: 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:51.018 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:51.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.017+0000 7f5b4a6cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:51.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.017+0000 7f5b4a6cf640 4 set_mon_vals no callback set 2024-03-25T19:46:51.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.018+0000 7f5b4a6cf640 10 monclient: shutdown 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.018+0000 7f5b4a6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.018+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/2746801105 >> v1:172.21.15.69:6790/0 conn(0x7f5b440b4d60 legacy=0x7f5b4413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.018+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/2746801105 shutdown_connections 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.018+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/2746801105 >> 172.21.15.69:0/2746801105 conn(0x7f5b440617a0 msgr2=0x7f5b44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.018+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/2746801105 shutdown_connections 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.018+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/2746801105 wait complete. 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.019+0000 7f5b4a6cf640 10 monclient: build_initial_monmap 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.019+0000 7f5b4a6cf640 10 monclient: monmap: 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:51.021 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.019+0000 7f5b4a6cf640 1 librados: starting msgr at 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.019+0000 7f5b4a6cf640 1 librados: starting objecter 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.019+0000 7f5b4a6cf640 1 Processor -- start 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 1 -- start start 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 1 librados: setting wanted keys 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 1 librados: calling monclient init 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 10 monclient: init 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f5b440b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f5b440b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f5b4414be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b44141bf0 con 0x7f5b440b1730 2024-03-25T19:46:51.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b44141ed0 con 0x7f5b4414be90 2024-03-25T19:46:51.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b441421b0 con 0x7f5b440b4d60 2024-03-25T19:46:51.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:51.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.020+0000 7f5b4a6cf640 10 monclient(hunting): authenticate will time out at 5223.289493s 2024-03-25T19:46:51.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.025+0000 7f5b43fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5b440b1730 0x7f5b441425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43372/0 (socket says 172.21.15.69:43372) 2024-03-25T19:46:51.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.025+0000 7f5b43fff640 1 -- 172.21.15.69:0/3468440495 learned_addr learned my addr 172.21.15.69:0/3468440495 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:51.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.028+0000 7f5b40ff9640 10 client.?.objecter ms_handle_connect 0x7f5b440b1730 2024-03-25T19:46:51.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.028+0000 7f5b40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:51.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.034+0000 7f5b40ff9640 10 client.?.objecter ms_handle_connect 0x7f5b4414be90 2024-03-25T19:46:51.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.034+0000 7f5b40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:51.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.036+0000 7f5b40ff9640 10 client.?.objecter ms_handle_connect 0x7f5b440b4d60 2024-03-25T19:46:51.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.036+0000 7f5b40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:51.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.043+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276800912 0 0) 0x7f5b44141bf0 con 0x7f5b440b1730 2024-03-25T19:46:51.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.043+0000 7f5b40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.043+0000 7f5b40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.043+0000 7f5b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.043+0000 7f5b40ff9640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:46:51.044 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.043+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30004eb0 con 0x7f5b440b1730 2024-03-25T19:46:51.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.047+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121460659 0 0) 0x7f5b441421b0 con 0x7f5b440b4d60 2024-03-25T19:46:51.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.047+0000 7f5b40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.047+0000 7f5b40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.047+0000 7f5b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.047+0000 7f5b40ff9640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:46:51.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.047+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b44141bf0 con 0x7f5b440b4d60 2024-03-25T19:46:51.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.050+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742769040 0 0) 0x7f5b44141ed0 con 0x7f5b4414be90 2024-03-25T19:46:51.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.050+0000 7f5b40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.050+0000 7f5b40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.050+0000 7f5b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.050+0000 7f5b40ff9640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:46:51.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.050+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b441421b0 con 0x7f5b4414be90 2024-03-25T19:46:51.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.059+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 708469578 0 0) 0x7f5b44141bf0 con 0x7f5b440b4d60 2024-03-25T19:46:51.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.059+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b44141ed0 con 0x7f5b440b4d60 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.060+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5b38001930 con 0x7f5b440b4d60 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.060+0000 7f5b40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.060+0000 7f5b40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.060+0000 7f5b40ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:51.060 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:51.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.060+0000 7f5b40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:51.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1938395995 0 0) 0x7f5b44141ed0 con 0x7f5b440b4d60 2024-03-25T19:46:51.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 >> v1:172.21.15.169:6789/0 conn(0x7f5b4414be90 legacy=0x7f5b441341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 >> v1:172.21.15.69:6789/0 conn(0x7f5b440b1730 legacy=0x7f5b441425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:51.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 1 monclient: found mon.c 2024-03-25T19:46:51.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:51.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:51.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b44140f00 con 0x7f5b440b4d60 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 5 monclient: authenticate success, global_id 4484 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 10 monclient: _renew_subs 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b441406f0 con 0x7f5b440b4d60 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 10 monclient: _renew_subs 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b44140380 con 0x7f5b440b4d60 2024-03-25T19:46:51.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.062+0000 7f5b4a6cf640 1 librados: init done 2024-03-25T19:46:51.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.064+0000 7f5b4a6cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:51.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.064+0000 7f5b4a6cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:51.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.064+0000 7f5b4a6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:51.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.064+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b441401f0 con 0x7f5b440b4d60 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.356+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b38003610 con 0x7f5b440b4d60 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.356+0000 7f5b40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.356+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5b38001610 con 0x7f5b440b4d60 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.356+0000 7f5b40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.356+0000 7f5b40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.356+0000 7f5b40ff9640 10 monclient: dump: 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:51.358 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:51.358 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:51.358 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.356+0000 7f5b427fc640 4 set_mon_vals no callback set 2024-03-25T19:46:51.358 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.357+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f5b38003b20 con 0x7f5b440b4d60 2024-03-25T19:46:51.358 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.358+0000 7f5b43fff640 10 monclient: get_auth_request con 0x7f5b3006ddf0 auth_method 0 2024-03-25T19:46:51.358 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.358+0000 7f5b40ff9640 10 client.4484.objecter ms_handle_connect 0x7f5b3006ddf0 2024-03-25T19:46:51.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.364+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f5b38053600 con 0x7f5b440b4d60 2024-03-25T19:46:51.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.364+0000 7f5b40ff9640 10 client.4484.objecter ms_dispatch 0x7f5b44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:51.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.364+0000 7f5b40ff9640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:51.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.364+0000 7f5b40ff9640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:51.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.364+0000 7f5b40ff9640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-25T19:46:51.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.364+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b3804f110 con 0x7f5b440b4d60 2024-03-25T19:46:51.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.364+0000 7f5b40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:51.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.364+0000 7f5b40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:51.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.594+0000 7f5b4a6cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.594+0000 7f5b4a6cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.594+0000 7f5b4a6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:46:51.595 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.594+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b440627b0 con 0x7f5b440b4d60 2024-03-25T19:46:51.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.630+0000 7f5b40ff9640 1 -- 172.21.15.69:0/3468440495 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f5b38051040 con 0x7f5b440b4d60 2024-03-25T19:46:51.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.630+0000 7f5b40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.630+0000 7f5b40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:51.630 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:46:51.630 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:51.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 10 librados: watch_flush enter 2024-03-25T19:46:51.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 10 librados: watch_flush exit 2024-03-25T19:46:51.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-25T19:46:51.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 10 client.4484.objecter successfully canceled tick 2024-03-25T19:46:51.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 >> v1:172.21.15.169:6800/3072548259 conn(0x7f5b3006ddf0 legacy=0x7f5b300702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 10 monclient: shutdown 2024-03-25T19:46:51.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:51.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 >> v1:172.21.15.69:6790/0 conn(0x7f5b440b4d60 legacy=0x7f5b44133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 shutdown_connections 2024-03-25T19:46:51.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.632+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 >> 172.21.15.69:0/3468440495 conn(0x7f5b440617a0 msgr2=0x7f5b4413c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:51.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.633+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 shutdown_connections 2024-03-25T19:46:51.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.633+0000 7f5b4a6cf640 1 -- 172.21.15.69:0/3468440495 wait complete. 2024-03-25T19:46:51.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:51.633+0000 7f5b4a6cf640 1 librados: shutdown 2024-03-25T19:46:51.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:58.451 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:58.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.561+0000 7fecedeba640 10 monclient: get_monmap_and_config 2024-03-25T19:46:58.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.561+0000 7fecedeba640 10 monclient: build_initial_monmap 2024-03-25T19:46:58.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.561+0000 7fecedeba640 10 monclient: monmap: 2024-03-25T19:46:58.561 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:46:58.562742+0000 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:46:58.562742+0000 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:46:58.562 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:58.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.562+0000 7fecedeba640 1 Processor -- start 2024-03-25T19:46:58.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.562+0000 7fecedeba640 1 -- start start 2024-03-25T19:46:58.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.562+0000 7fecedeba640 10 monclient: init 2024-03-25T19:46:58.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 10 monclient(hunting): picked mon.noname-a con 0x7fece80b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 10 monclient(hunting): picked mon.noname-c con 0x7fece80b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 10 monclient(hunting): picked mon.noname-b con 0x7fece814be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80af940 con 0x7fece80b4d60 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80b0a70 con 0x7fece80b1730 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80b0d50 con 0x7fece814be90 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:58.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.563+0000 7fecedeba640 10 monclient(hunting): authenticate will time out at 5230.832000s 2024-03-25T19:46:58.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.564+0000 7fece6ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fece80b1730 0x7fece80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:51096/0 (socket says 172.21.15.69:51096) 2024-03-25T19:46:58.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.564+0000 7fece6ffd640 1 -- 172.21.15.69:0/4186641517 learned_addr learned my addr 172.21.15.69:0/4186641517 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:58.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.632+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771188243 0 0) 0x7fece80af940 con 0x7fece80b4d60 2024-03-25T19:46:58.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.632+0000 7fece67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.632+0000 7fece67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.632+0000 7fece67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.632+0000 7fece67fc640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:46:58.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.632+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd0003520 con 0x7fece80b4d60 2024-03-25T19:46:58.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686833432 0 0) 0x7fecd0003520 con 0x7fece80b4d60 2024-03-25T19:46:58.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 >> v1:172.21.15.69:6790/0 conn(0x7fece814be90 legacy=0x7fece814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 >> v1:172.21.15.169:6789/0 conn(0x7fece80b1730 legacy=0x7fece80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 1 monclient: found mon.noname-a 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece80b1030 con 0x7fece80b4d60 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fecd8001690 con 0x7fece80b4d60 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fece67fc640 10 monclient: dump: 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:58.654 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fecedeba640 5 monclient: authenticate success, global_id 4506 2024-03-25T19:46:58.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.653+0000 7fecedeba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.671+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecd80017f0 con 0x7fece80b4d60 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.671+0000 7fece67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.671+0000 7fece67fc640 1 -- 172.21.15.69:0/4186641517 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fecd8001c00 con 0x7fece80b4d60 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.671+0000 7fecedeba640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.671+0000 7fecedeba640 4 set_mon_vals no callback set 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.671+0000 7fece67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.671+0000 7fece67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.671+0000 7fece67fc640 10 monclient: dump: 2024-03-25T19:46:58.671 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:58.672 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.672+0000 7fecedeba640 10 monclient: shutdown 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.672+0000 7fecedeba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.672+0000 7fecedeba640 1 -- 172.21.15.69:0/4186641517 >> v1:172.21.15.69:6789/0 conn(0x7fece80b4d60 legacy=0x7fece813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.672+0000 7fecedeba640 1 -- 172.21.15.69:0/4186641517 shutdown_connections 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.672+0000 7fecedeba640 1 -- 172.21.15.69:0/4186641517 >> 172.21.15.69:0/4186641517 conn(0x7fece80617a0 msgr2=0x7fece8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.672+0000 7fecedeba640 1 -- 172.21.15.69:0/4186641517 shutdown_connections 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.672+0000 7fecedeba640 1 -- 172.21.15.69:0/4186641517 wait complete. 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.672+0000 7fecedeba640 10 monclient: build_initial_monmap 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.673+0000 7fecedeba640 10 monclient: monmap: 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:58.674 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.673+0000 7fecedeba640 1 librados: starting msgr at 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.673+0000 7fecedeba640 1 librados: starting objecter 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.673+0000 7fecedeba640 1 Processor -- start 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.673+0000 7fecedeba640 1 -- start start 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.673+0000 7fecedeba640 1 librados: setting wanted keys 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.673+0000 7fecedeba640 1 librados: calling monclient init 2024-03-25T19:46:58.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.673+0000 7fecedeba640 10 monclient: init 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 10 monclient(hunting): picked mon.noname-c con 0x7fece80b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 10 monclient(hunting): picked mon.noname-a con 0x7fece80b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 10 monclient(hunting): picked mon.noname-b con 0x7fece814be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece813bde0 con 0x7fece80b4d60 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece8140f50 con 0x7fece814be90 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece81410e0 con 0x7fece80b1730 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:58.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.674+0000 7fecedeba640 10 monclient(hunting): authenticate will time out at 5230.943321s 2024-03-25T19:46:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.681+0000 7fece6ffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fece80b4d60 0x7fece81d7d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48194/0 (socket says 172.21.15.69:48194) 2024-03-25T19:46:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.681+0000 7fece6ffd640 1 -- 172.21.15.69:0/2591412605 learned_addr learned my addr 172.21.15.69:0/2591412605 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:46:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.682+0000 7feccffff640 10 client.?.objecter ms_handle_connect 0x7fece80b4d60 2024-03-25T19:46:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.682+0000 7feccffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.689+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023500941 0 0) 0x7fece813bde0 con 0x7fece80b4d60 2024-03-25T19:46:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.689+0000 7feccffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.689+0000 7feccffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.689+0000 7feccffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.689+0000 7feccffff640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:46:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.689+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd4004eb0 con 0x7fece80b4d60 2024-03-25T19:46:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.705+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1566621674 0 0) 0x7fecd4004eb0 con 0x7fece80b4d60 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.705+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fece813bde0 con 0x7fece80b4d60 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.705+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fecdc0018b0 con 0x7fece80b4d60 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.705+0000 7feccffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.705+0000 7feccffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.705+0000 7feccffff640 10 monclient(hunting): dump: 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:58.706 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:58.707 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:58.707 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:58.707 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.705+0000 7feccffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:58.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.708+0000 7feccffff640 10 client.?.objecter ms_handle_connect 0x7fece80b1730 2024-03-25T19:46:58.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.708+0000 7feccffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.711 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.711+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141032423 0 0) 0x7fece81410e0 con 0x7fece80b1730 2024-03-25T19:46:58.711 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.711+0000 7feccffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.711 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.711+0000 7feccffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.711 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.711+0000 7feccffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.711 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.711+0000 7feccffff640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:46:58.711 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.711+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd4004eb0 con 0x7fece80b1730 2024-03-25T19:46:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 26690224 0 0) 0x7fece813bde0 con 0x7fece80b4d60 2024-03-25T19:46:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 >> v1:172.21.15.69:6790/0 conn(0x7fece80b1730 legacy=0x7fece8140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 >> v1:172.21.15.169:6789/0 conn(0x7fece814be90 legacy=0x7fece8142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 1 monclient: found mon.a 2024-03-25T19:46:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece81d8450 con 0x7fece80b4d60 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 10 monclient: _finish_auth 0 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7feccffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7fecedeba640 5 monclient: authenticate success, global_id 4509 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7fecedeba640 10 monclient: _renew_subs 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7fecedeba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece81d86a0 con 0x7fece80b4d60 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.717+0000 7fecedeba640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.718+0000 7fecedeba640 10 monclient: _renew_subs 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.718+0000 7fecedeba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.718+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fece81426d0 con 0x7fece80b4d60 2024-03-25T19:46:58.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.718+0000 7fecedeba640 1 librados: init done 2024-03-25T19:46:58.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.718+0000 7fecedeba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:58.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.718+0000 7fecedeba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:58.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.718+0000 7fecedeba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:58.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:58.718+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece81401f0 con 0x7fece80b4d60 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.059+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecdc001a10 con 0x7fece80b4d60 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.059+0000 7feccffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.059+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fecdc004a00 con 0x7fece80b4d60 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.059+0000 7feccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.059+0000 7feccffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.059+0000 7feccffff640 10 monclient: dump: 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:59.060 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.059+0000 7fece5ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.060+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fecdc004c20 con 0x7fece80b4d60 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.060+0000 7fece77fe640 10 monclient: get_auth_request con 0x7fecd406dc90 auth_method 0 2024-03-25T19:46:59.061 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.061+0000 7feccffff640 10 client.4509.objecter ms_handle_connect 0x7fecd406dc90 2024-03-25T19:46:59.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.065+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fecdc054930 con 0x7fece80b4d60 2024-03-25T19:46:59.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.065+0000 7feccffff640 10 client.4509.objecter ms_dispatch 0x7fece8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:59.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.065+0000 7feccffff640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:59.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.065+0000 7feccffff640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:59.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.066+0000 7feccffff640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T19:46:59.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.066+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecdc050270 con 0x7fece80b4d60 2024-03-25T19:46:59.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.066+0000 7feccffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:59.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.066+0000 7feccffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:59.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.314+0000 7fecedeba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:59.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.314+0000 7fecedeba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:59.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.314+0000 7fecedeba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:46:59.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.314+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fece81d7a60 con 0x7fece80b4d60 2024-03-25T19:46:59.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.337+0000 7feccffff640 1 -- 172.21.15.69:0/2591412605 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fece81d7a60 con 0x7fece80b4d60 2024-03-25T19:46:59.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.337+0000 7feccffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:59.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.337+0000 7feccffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:59.337 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:46:59.337 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:59.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 10 librados: watch_flush enter 2024-03-25T19:46:59.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 10 librados: watch_flush exit 2024-03-25T19:46:59.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T19:46:59.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 10 client.4509.objecter successfully canceled tick 2024-03-25T19:46:59.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 >> v1:172.21.15.169:6800/3072548259 conn(0x7fecd406dc90 legacy=0x7fecd4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:59.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 10 monclient: shutdown 2024-03-25T19:46:59.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:59.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 >> v1:172.21.15.69:6789/0 conn(0x7fece80b4d60 legacy=0x7fece81d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:59.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.339+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 shutdown_connections 2024-03-25T19:46:59.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.340+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 >> 172.21.15.69:0/2591412605 conn(0x7fece80617a0 msgr2=0x7fece80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:59.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.340+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 shutdown_connections 2024-03-25T19:46:59.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.340+0000 7fecedeba640 1 -- 172.21.15.69:0/2591412605 wait complete. 2024-03-25T19:46:59.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:46:59.340+0000 7fecedeba640 1 librados: shutdown 2024-03-25T19:46:59.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:06.157 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:06.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.269+0000 7f0da380e640 10 monclient: get_monmap_and_config 2024-03-25T19:47:06.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.269+0000 7f0da380e640 10 monclient: build_initial_monmap 2024-03-25T19:47:06.269 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.269+0000 7f0da380e640 10 monclient: monmap: 2024-03-25T19:47:06.269 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:06.269 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:06.269 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:47:06.270115+0000 2024-03-25T19:47:06.269 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:47:06.270115+0000 2024-03-25T19:47:06.270 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:06.270 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:06.270 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:06.270 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:47:06.270 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:47:06.270 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:06.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.270+0000 7f0da380e640 1 Processor -- start 2024-03-25T19:47:06.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.270+0000 7f0da380e640 1 -- start start 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.270+0000 7f0da380e640 10 monclient: init 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 10 monclient(hunting): picked mon.noname-a con 0x7f0d9c0a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 10 monclient(hunting): picked mon.noname-c con 0x7f0d9c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 10 monclient(hunting): picked mon.noname-b con 0x7f0d9c1430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c142520 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c0b28a0 con 0x7f0d9c13f710 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c0b2a30 con 0x7f0d9c1430c0 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:06.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.271+0000 7f0da380e640 10 monclient(hunting): authenticate will time out at 5238.539202s 2024-03-25T19:47:06.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.282+0000 7f0da0da9640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f0d9c13f710 0x7f0d9c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:40434/0 (socket says 172.21.15.69:40434) 2024-03-25T19:47:06.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.282+0000 7f0da0da9640 1 -- 172.21.15.69:0/1266617033 learned_addr learned my addr 172.21.15.69:0/1266617033 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:06.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166356531 0 0) 0x7f0d9c0b2a30 con 0x7f0d9c1430c0 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d88003520 con 0x7f0d9c1430c0 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574196292 0 0) 0x7f0d9c0b28a0 con 0x7f0d9c13f710 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:47:06.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.299+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c0b2a30 con 0x7f0d9c13f710 2024-03-25T19:47:06.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.303+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685210005 0 0) 0x7f0d9c142520 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.303 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.303+0000 7f0d93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.303 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.303+0000 7f0d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.303 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.303+0000 7f0d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.303 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.303+0000 7f0d93fff640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:47:06.303 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.303+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c0b28a0 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.312 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751205581 0 0) 0x7f0d9c0b2a30 con 0x7f0d9c13f710 2024-03-25T19:47:06.312 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 >> v1:172.21.15.69:6790/0 conn(0x7f0d9c1430c0 legacy=0x7f0d9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.312 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 >> v1:172.21.15.69:6789/0 conn(0x7f0d9c0a98f0 legacy=0x7f0d9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 1 monclient: found mon.noname-c 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c0b44c0 con 0x7f0d9c13f710 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0d8c001670 con 0x7f0d9c13f710 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0da380e640 5 monclient: authenticate success, global_id 4480 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0da380e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0d93fff640 10 monclient: dump: 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:06.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.313+0000 7f0da380e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:06.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.321+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d8c0017d0 con 0x7f0d9c13f710 2024-03-25T19:47:06.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.321+0000 7f0d93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:06.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.321+0000 7f0d93fff640 1 -- 172.21.15.69:0/1266617033 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0d8c001be0 con 0x7f0d9c13f710 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.321+0000 7f0da380e640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.321+0000 7f0da380e640 4 set_mon_vals no callback set 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.321+0000 7f0d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.321+0000 7f0d93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.321+0000 7f0d93fff640 10 monclient: dump: 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:06.323 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.322+0000 7f0da380e640 10 monclient: shutdown 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.322+0000 7f0da380e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.322+0000 7f0da380e640 1 -- 172.21.15.69:0/1266617033 >> v1:172.21.15.169:6789/0 conn(0x7f0d9c13f710 legacy=0x7f0d9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.322+0000 7f0da380e640 1 -- 172.21.15.69:0/1266617033 shutdown_connections 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.322+0000 7f0da380e640 1 -- 172.21.15.69:0/1266617033 >> 172.21.15.69:0/1266617033 conn(0x7f0d9c0b7ea0 msgr2=0x7f0d9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.322+0000 7f0da380e640 1 -- 172.21.15.69:0/1266617033 shutdown_connections 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.322+0000 7f0da380e640 1 -- 172.21.15.69:0/1266617033 wait complete. 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.323+0000 7f0da380e640 10 monclient: build_initial_monmap 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.323+0000 7f0da380e640 10 monclient: monmap: 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:06.324 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:47:06.325 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:47:06.325 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:06.325 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:47:06.325 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:06.325 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:47:06.325 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:47:06.325 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:06.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 1 librados: starting msgr at 2024-03-25T19:47:06.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 1 librados: starting objecter 2024-03-25T19:47:06.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 1 Processor -- start 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 1 -- start start 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 1 librados: setting wanted keys 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 1 librados: calling monclient init 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 10 monclient: init 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.324+0000 7f0da380e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.325+0000 7f0da380e640 10 monclient(hunting): picked mon.noname-c con 0x7f0d9c0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.325+0000 7f0da380e640 10 monclient(hunting): picked mon.noname-b con 0x7f0d9c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.325+0000 7f0da380e640 10 monclient(hunting): picked mon.noname-a con 0x7f0d9c1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.325+0000 7f0da380e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c1d7a90 con 0x7f0d9c1430c0 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.325+0000 7f0da380e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c0b1700 con 0x7f0d9c13f710 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.325+0000 7f0da380e640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c0b13d0 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.325+0000 7f0da380e640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:06.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.325+0000 7f0da380e640 10 monclient(hunting): authenticate will time out at 5238.592698s 2024-03-25T19:47:06.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.328+0000 7f0da1dab640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0d9c1430c0 0x7f0d9c14d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:40718/0 (socket says 172.21.15.69:40718) 2024-03-25T19:47:06.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.328+0000 7f0da1dab640 1 -- 172.21.15.69:0/61084709 learned_addr learned my addr 172.21.15.69:0/61084709 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:06.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.329+0000 7f0d91ffb640 10 client.?.objecter ms_handle_connect 0x7f0d9c1430c0 2024-03-25T19:47:06.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.329+0000 7f0d91ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:06.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.331+0000 7f0d91ffb640 10 client.?.objecter ms_handle_connect 0x7f0d9c0a98f0 2024-03-25T19:47:06.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.331+0000 7f0d91ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:06.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.341+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665545651 0 0) 0x7f0d9c1d7a90 con 0x7f0d9c1430c0 2024-03-25T19:47:06.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.341+0000 7f0d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.341+0000 7f0d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.341+0000 7f0d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.341+0000 7f0d91ffb640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:47:06.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.341+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d80004eb0 con 0x7f0d9c1430c0 2024-03-25T19:47:06.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.346+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342082324 0 0) 0x7f0d9c0b13d0 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.346+0000 7f0d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.346+0000 7f0d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.346+0000 7f0d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.346+0000 7f0d91ffb640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:47:06.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.346+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c1d7a90 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.352 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.353+0000 7f0d91ffb640 10 client.?.objecter ms_handle_connect 0x7f0d9c13f710 2024-03-25T19:47:06.352 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.353+0000 7f0d91ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:06.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.354+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1110004697 0 0) 0x7f0d80004eb0 con 0x7f0d9c1430c0 2024-03-25T19:47:06.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.354+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d9c0b13d0 con 0x7f0d9c1430c0 2024-03-25T19:47:06.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.354+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0d94001c30 con 0x7f0d9c1430c0 2024-03-25T19:47:06.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.354+0000 7f0d91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.354+0000 7f0d91ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.354+0000 7f0d91ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:06.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.354+0000 7f0d91ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:06.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.356+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2364378656 0 0) 0x7f0d9c1d7a90 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.356+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d80004eb0 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.356+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0d980018b0 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.356+0000 7f0d91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.356+0000 7f0d91ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:06.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.356+0000 7f0d91ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:06.355 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:06.356 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:06.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.365+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804475419 0 0) 0x7f0d9c0b1700 con 0x7f0d9c13f710 2024-03-25T19:47:06.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.365+0000 7f0d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.365+0000 7f0d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.365+0000 7f0d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.365+0000 7f0d91ffb640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:47:06.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.365+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c1d7a90 con 0x7f0d9c13f710 2024-03-25T19:47:06.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.385+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189432735 0 0) 0x7f0d9c1d7a90 con 0x7f0d9c13f710 2024-03-25T19:47:06.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.385+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d9c0b1700 con 0x7f0d9c13f710 2024-03-25T19:47:06.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.385+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0d8c001ae0 con 0x7f0d9c13f710 2024-03-25T19:47:06.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.385+0000 7f0d91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.385+0000 7f0d91ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:06.384 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.385+0000 7f0d91ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:06.384 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:06.385 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:06.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3579259672 0 0) 0x7f0d80004eb0 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 >> v1:172.21.15.169:6789/0 conn(0x7f0d9c13f710 legacy=0x7f0d9c1d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 >> v1:172.21.15.69:6789/0 conn(0x7f0d9c1430c0 legacy=0x7f0d9c14d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:47:06.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 1 monclient: found mon.c 2024-03-25T19:47:06.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:06.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c14eb50 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 10 monclient: _finish_auth 0 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0d91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0da380e640 5 monclient: authenticate success, global_id 4496 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0da380e640 10 monclient: _renew_subs 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0da380e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.387+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9c14dc50 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.388+0000 7f0da380e640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.388+0000 7f0da380e640 10 monclient: _renew_subs 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.388+0000 7f0da380e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.388+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9c14e290 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.388+0000 7f0da380e640 1 librados: init done 2024-03-25T19:47:06.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.389+0000 7f0da380e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:06.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.389+0000 7f0da380e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:06.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.389+0000 7f0da380e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:06.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.389+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9c0b1560 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.707+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d98003610 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.707+0000 7f0d91ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:06.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.707+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0d98003800 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.707+0000 7f0d91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.707+0000 7f0d91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.707+0000 7f0d91ffb640 10 monclient: dump: 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:06.707 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:06.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.707+0000 7f0d937fe640 4 set_mon_vals no callback set 2024-03-25T19:47:06.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.708+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f0d98003a20 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.708+0000 7f0da0da9640 10 monclient: get_auth_request con 0x7f0d8006e120 auth_method 0 2024-03-25T19:47:06.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.709+0000 7f0d91ffb640 10 client.4496.objecter ms_handle_connect 0x7f0d8006e120 2024-03-25T19:47:06.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.713+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f0d980533d0 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.713+0000 7f0d91ffb640 10 client.4496.objecter ms_dispatch 0x7f0d9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:06.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.713+0000 7f0d91ffb640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:06.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.713+0000 7f0d91ffb640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:06.713 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.713+0000 7f0d91ffb640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T19:47:06.713 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.713+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d9804eee0 con 0x7f0d9c0a98f0 2024-03-25T19:47:06.713 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.713+0000 7f0d91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:06.713 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.713+0000 7f0d91ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:06.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.930+0000 7f0da380e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:06.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.930+0000 7f0da380e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:06.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.930+0000 7f0da380e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:06.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:06.930+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d9c0b0d00 con 0x7f0d9c0a98f0 2024-03-25T19:47:07.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.024+0000 7f0d91ffb640 1 -- 172.21.15.69:0/61084709 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f0d98050e10 con 0x7f0d9c0a98f0 2024-03-25T19:47:07.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.024+0000 7f0d91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:07.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.024+0000 7f0d91ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:07.024 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:47:07.024 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:07.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.026+0000 7f0da380e640 10 librados: watch_flush enter 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.026+0000 7f0da380e640 10 librados: watch_flush exit 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.026+0000 7f0da380e640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.026+0000 7f0da380e640 10 client.4496.objecter successfully canceled tick 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 >> v1:172.21.15.169:6800/3072548259 conn(0x7f0d8006e120 legacy=0x7f0d800705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 10 monclient: shutdown 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 >> v1:172.21.15.69:6790/0 conn(0x7f0d9c0a98f0 legacy=0x7f0d9c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 shutdown_connections 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 >> 172.21.15.69:0/61084709 conn(0x7f0d9c0b7ea0 msgr2=0x7f0d9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 shutdown_connections 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 1 -- 172.21.15.69:0/61084709 wait complete. 2024-03-25T19:47:07.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:07.027+0000 7f0da380e640 1 librados: shutdown 2024-03-25T19:47:07.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:08.078 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:46.042 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:08.128 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:46.092 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:08.161 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:46.125 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:08.183 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:46.137 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:08.185 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:46.140 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:08.194 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:46.158 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:08.203 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:46.167 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:08.222 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:46.176 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:08.242 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:46.206 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.842 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:13.956 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.956+0000 7f9ad7354640 10 monclient: get_monmap_and_config 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.956+0000 7f9ad7354640 10 monclient: build_initial_monmap 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.956+0000 7f9ad7354640 10 monclient: monmap: 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:47:13.957693+0000 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:47:13.957693+0000 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:47:13.957 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:13.959 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.957+0000 7f9ad7354640 1 Processor -- start 2024-03-25T19:47:13.959 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.957+0000 7f9ad7354640 1 -- start start 2024-03-25T19:47:13.959 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.957+0000 7f9ad7354640 10 monclient: init 2024-03-25T19:47:13.959 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:13.959 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:13.959 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad00b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:13.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:13.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:13.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad00af940 con 0x7f9ad00b4d60 2024-03-25T19:47:13.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad00b0a70 con 0x7f9ad00b1730 2024-03-25T19:47:13.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad00b0d50 con 0x7f9ad014be90 2024-03-25T19:47:13.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:13.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.958+0000 7f9ad7354640 10 monclient(hunting): authenticate will time out at 5246.227247s 2024-03-25T19:47:13.970 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.970+0000 7f9ad58f1640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f9ad014be90 0x7f9ad014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:34298/0 (socket says 172.21.15.69:34298) 2024-03-25T19:47:13.970 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.970+0000 7f9ad58f1640 1 -- 172.21.15.69:0/1054564624 learned_addr learned my addr 172.21.15.69:0/1054564624 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:13.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.986+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341276255 0 0) 0x7f9ad00af940 con 0x7f9ad00b4d60 2024-03-25T19:47:13.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.986+0000 7f9ac7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.986+0000 7f9ac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.986+0000 7f9ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.986+0000 7f9ac7fff640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:47:13.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.986+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9abc003520 con 0x7f9ad00b4d60 2024-03-25T19:47:13.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.992+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937147222 0 0) 0x7f9ad00b0d50 con 0x7f9ad014be90 2024-03-25T19:47:13.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.992+0000 7f9ac7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.992+0000 7f9ac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.992+0000 7f9ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.992+0000 7f9ac7fff640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:47:13.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:13.992+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad00af940 con 0x7f9ad014be90 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.000+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2538169968 0 0) 0x7f9ad00af940 con 0x7f9ad014be90 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.000+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 >> v1:172.21.15.169:6789/0 conn(0x7f9ad00b1730 legacy=0x7f9ad00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.000+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 >> v1:172.21.15.69:6789/0 conn(0x7f9ad00b4d60 legacy=0x7f9ad013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.000+0000 7f9ac7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.000+0000 7f9ac7fff640 1 monclient: found mon.noname-b 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.000+0000 7f9ac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.000+0000 7f9ac7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.000+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad00b1030 con 0x7f9ad014be90 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ac7fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ac7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:14.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9ac80019f0 con 0x7f9ad014be90 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ad7354640 5 monclient: authenticate success, global_id 4499 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ad7354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ac7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ac7fff640 10 monclient: dump: 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:14.002 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:14.003 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:14.003 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:14.003 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:14.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.001+0000 7f9ad7354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:14.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.012+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ac8001b50 con 0x7f9ad014be90 2024-03-25T19:47:14.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.012+0000 7f9ac7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:14.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.012+0000 7f9ac7fff640 1 -- 172.21.15.69:0/1054564624 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9ac8001fa0 con 0x7f9ad014be90 2024-03-25T19:47:14.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.012+0000 7f9ad7354640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:14.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.012+0000 7f9ad7354640 4 set_mon_vals no callback set 2024-03-25T19:47:14.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.012+0000 7f9ac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.012+0000 7f9ac7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.012+0000 7f9ac7fff640 10 monclient: dump: 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:14.013 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:14.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.013+0000 7f9ad7354640 10 monclient: shutdown 2024-03-25T19:47:14.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.013+0000 7f9ad7354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:14.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.013+0000 7f9ad7354640 1 -- 172.21.15.69:0/1054564624 >> v1:172.21.15.69:6790/0 conn(0x7f9ad014be90 legacy=0x7f9ad014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.013+0000 7f9ad7354640 1 -- 172.21.15.69:0/1054564624 shutdown_connections 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.013+0000 7f9ad7354640 1 -- 172.21.15.69:0/1054564624 >> 172.21.15.69:0/1054564624 conn(0x7f9ad00617a0 msgr2=0x7f9ad0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.013+0000 7f9ad7354640 1 -- 172.21.15.69:0/1054564624 shutdown_connections 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.013+0000 7f9ad7354640 1 -- 172.21.15.69:0/1054564624 wait complete. 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 10 monclient: build_initial_monmap 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 10 monclient: monmap: 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:47:14.016 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 1 librados: starting msgr at 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 1 librados: starting objecter 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 1 Processor -- start 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 1 -- start start 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 1 librados: setting wanted keys 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 1 librados: calling monclient init 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.014+0000 7f9ad7354640 10 monclient: init 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad00b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:14.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad014be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:14.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad0143220 con 0x7f9ad014be90 2024-03-25T19:47:14.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad01d7870 con 0x7f9ad00b1730 2024-03-25T19:47:14.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad013d490 con 0x7f9ad00b4d60 2024-03-25T19:47:14.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:14.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.015+0000 7f9ad7354640 10 monclient(hunting): authenticate will time out at 5246.283921s 2024-03-25T19:47:14.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.026+0000 7f9ad50f0640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f9ad00b1730 0x7f9ad0141bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:44422/0 (socket says 172.21.15.69:44422) 2024-03-25T19:47:14.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.026+0000 7f9ad50f0640 1 -- 172.21.15.69:0/2023663272 learned_addr learned my addr 172.21.15.69:0/2023663272 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:14.026 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.026+0000 7f9ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f9ad00b1730 2024-03-25T19:47:14.027 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.027+0000 7f9ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:14.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.029+0000 7f9ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f9ad014be90 2024-03-25T19:47:14.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.029+0000 7f9ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:14.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.036+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923390770 0 0) 0x7f9ad01d7870 con 0x7f9ad00b1730 2024-03-25T19:47:14.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.036+0000 7f9ac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:14.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.036+0000 7f9ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:14.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.036+0000 7f9ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:14.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.036+0000 7f9ac5ffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:47:14.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.036+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab4004eb0 con 0x7f9ad00b1730 2024-03-25T19:47:14.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.042+0000 7f9ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f9ad00b4d60 2024-03-25T19:47:14.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.042+0000 7f9ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:14.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.050+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708851123 0 0) 0x7f9ad013d490 con 0x7f9ad00b4d60 2024-03-25T19:47:14.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.050+0000 7f9ac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:14.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.050+0000 7f9ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:14.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.050+0000 7f9ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:14.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.050+0000 7f9ac5ffb640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:47:14.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.050+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad01d7870 con 0x7f9ad00b4d60 2024-03-25T19:47:14.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.052+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142368899 0 0) 0x7f9ad0143220 con 0x7f9ad014be90 2024-03-25T19:47:14.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.052+0000 7f9ac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:14.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.052+0000 7f9ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:14.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.052+0000 7f9ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:14.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.052+0000 7f9ac5ffb640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:47:14.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.052+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad013d490 con 0x7f9ad014be90 2024-03-25T19:47:14.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.058+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2779010302 0 0) 0x7f9ab4004eb0 con 0x7f9ad00b1730 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.058+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ad0143220 con 0x7f9ad00b1730 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.058+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9acc001a10 con 0x7f9ad00b1730 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.058+0000 7f9ac5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.058+0000 7f9ac5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.058+0000 7f9ac5ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:14.059 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:14.060 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:14.060 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:14.060 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:14.060 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.058+0000 7f9ac5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:14.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.070+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224505316 0 0) 0x7f9ad01d7870 con 0x7f9ad00b4d60 2024-03-25T19:47:14.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.070+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab4004eb0 con 0x7f9ad00b4d60 2024-03-25T19:47:14.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.070+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9ac00018a0 con 0x7f9ad00b4d60 2024-03-25T19:47:14.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.070+0000 7f9ac5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.070+0000 7f9ac5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.070+0000 7f9ac5ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:14.071 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1060898579 0 0) 0x7f9ad0143220 con 0x7f9ad00b1730 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 >> v1:172.21.15.69:6790/0 conn(0x7f9ad00b4d60 legacy=0x7f9ad01d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 >> v1:172.21.15.69:6789/0 conn(0x7f9ad014be90 legacy=0x7f9ad0132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 1 monclient: found mon.b 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad01422c0 con 0x7f9ad00b1730 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ac5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 10 monclient: _renew_subs 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad0140ea0 con 0x7f9ad00b1730 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 10 monclient: _renew_subs 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad01d8680 con 0x7f9ad00b1730 2024-03-25T19:47:14.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.072+0000 7f9ad7354640 1 librados: init done 2024-03-25T19:47:14.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.073+0000 7f9ad7354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:14.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.073+0000 7f9ad7354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:14.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.073+0000 7f9ad7354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:14.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.073+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad01d7a00 con 0x7f9ad00b1730 2024-03-25T19:47:14.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.363+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9acc001c50 con 0x7f9ad00b1730 2024-03-25T19:47:14.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.363+0000 7f9ac5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:14.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.363+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9acc003a60 con 0x7f9ad00b1730 2024-03-25T19:47:14.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.363+0000 7f9ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.363+0000 7f9ac5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.363+0000 7f9ac5ffb640 10 monclient: dump: 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:14.364 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.363+0000 7f9ac77fe640 4 set_mon_vals no callback set 2024-03-25T19:47:14.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.364+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f9acc003c40 con 0x7f9ad00b1730 2024-03-25T19:47:14.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.365+0000 7f9ad48ef640 10 monclient: get_auth_request con 0x7f9ab406df70 auth_method 0 2024-03-25T19:47:14.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.365+0000 7f9ac5ffb640 10 client.4489.objecter ms_handle_connect 0x7f9ab406df70 2024-03-25T19:47:14.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.369+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f9acc054950 con 0x7f9ad00b1730 2024-03-25T19:47:14.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.369+0000 7f9ac5ffb640 10 client.4489.objecter ms_dispatch 0x7f9ad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:14.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.369+0000 7f9ac5ffb640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:14.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.369+0000 7f9ac5ffb640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:14.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.369+0000 7f9ac5ffb640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T19:47:14.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.369+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9acc050460 con 0x7f9ad00b1730 2024-03-25T19:47:14.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.369+0000 7f9ac5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:14.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.369+0000 7f9ac5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:14.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.616+0000 7f9ad7354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.616+0000 7f9ad7354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.616+0000 7f9ad7354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:14.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.616+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ad01424d0 con 0x7f9ad00b1730 2024-03-25T19:47:14.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.656+0000 7f9ac5ffb640 1 -- 172.21.15.69:0/2023663272 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f9acc052390 con 0x7f9ad00b1730 2024-03-25T19:47:14.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.656+0000 7f9ac5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.656+0000 7f9ac5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:14.656 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:47:14.657 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:14.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.658+0000 7f9ad7354640 10 librados: watch_flush enter 2024-03-25T19:47:14.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.658+0000 7f9ad7354640 10 librados: watch_flush exit 2024-03-25T19:47:14.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.658+0000 7f9ad7354640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T19:47:14.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.658+0000 7f9ad7354640 10 client.4489.objecter successfully canceled tick 2024-03-25T19:47:14.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 >> v1:172.21.15.169:6800/3072548259 conn(0x7f9ab406df70 legacy=0x7f9ab4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 10 monclient: shutdown 2024-03-25T19:47:14.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:14.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 >> v1:172.21.15.169:6789/0 conn(0x7f9ad00b1730 legacy=0x7f9ad0141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 shutdown_connections 2024-03-25T19:47:14.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 >> 172.21.15.69:0/2023663272 conn(0x7f9ad00617a0 msgr2=0x7f9ad00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:14.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 shutdown_connections 2024-03-25T19:47:14.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 1 -- 172.21.15.69:0/2023663272 wait complete. 2024-03-25T19:47:14.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:14.659+0000 7f9ad7354640 1 librados: shutdown 2024-03-25T19:47:14.669 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:21.476 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:21.596 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.596+0000 7f19ecc25640 10 monclient: get_monmap_and_config 2024-03-25T19:47:21.596 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.596+0000 7f19ecc25640 10 monclient: build_initial_monmap 2024-03-25T19:47:21.596 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.596+0000 7f19ecc25640 10 monclient: monmap: 2024-03-25T19:47:21.596 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:21.596 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:21.596 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:47:21.597240+0000 2024-03-25T19:47:21.596 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:47:21.597240+0000 2024-03-25T19:47:21.596 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:21.597 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:21.597 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:21.597 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:47:21.597 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:47:21.597 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:21.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.596+0000 7f19ecc25640 1 Processor -- start 2024-03-25T19:47:21.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.596+0000 7f19ecc25640 1 -- start start 2024-03-25T19:47:21.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.596+0000 7f19ecc25640 10 monclient: init 2024-03-25T19:47:21.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:21.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:21.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 10 monclient(hunting): picked mon.noname-a con 0x7f19e806c380 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:21.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 10 monclient(hunting): picked mon.noname-c con 0x7f19e806d670 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:21.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 10 monclient(hunting): picked mon.noname-b con 0x7f19e80af8f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:21.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e806c9a0 con 0x7f19e806c380 2024-03-25T19:47:21.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e80ada90 con 0x7f19e806d670 2024-03-25T19:47:21.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e80b0d10 con 0x7f19e80af8f0 2024-03-25T19:47:21.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:21.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.597+0000 7f19ecc25640 10 monclient(hunting): authenticate will time out at 5253.866597s 2024-03-25T19:47:21.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.606+0000 7f19e5d9b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f19e806d670 0x7f19e80ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:60080/0 (socket says 172.21.15.69:60080) 2024-03-25T19:47:21.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.606+0000 7f19e5d9b640 1 -- 172.21.15.69:0/4284519624 learned_addr learned my addr 172.21.15.69:0/4284519624 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:21.637 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.637+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114156040 0 0) 0x7f19e80ada90 con 0x7f19e806d670 2024-03-25T19:47:21.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.637+0000 7f19e559a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.637+0000 7f19e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.637+0000 7f19e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.637+0000 7f19e559a640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:47:21.638 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.637+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d0003520 con 0x7f19e806d670 2024-03-25T19:47:21.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.639+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289500254 0 0) 0x7f19e806c9a0 con 0x7f19e806c380 2024-03-25T19:47:21.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.639+0000 7f19e559a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.639+0000 7f19e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.639+0000 7f19e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.639+0000 7f19e559a640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:47:21.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.639+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e80ada90 con 0x7f19e806c380 2024-03-25T19:47:21.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.646+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338106209 0 0) 0x7f19e80b0d10 con 0x7f19e80af8f0 2024-03-25T19:47:21.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.646+0000 7f19e559a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.646+0000 7f19e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.646+0000 7f19e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.646+0000 7f19e559a640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:47:21.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.646+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e806c9a0 con 0x7f19e80af8f0 2024-03-25T19:47:21.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 582524715 0 0) 0x7f19e80ada90 con 0x7f19e806c380 2024-03-25T19:47:21.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 >> v1:172.21.15.69:6790/0 conn(0x7f19e80af8f0 legacy=0x7f19e80aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 >> v1:172.21.15.169:6789/0 conn(0x7f19e806d670 legacy=0x7f19e80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 10 monclient: _finish_hunting 0 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 1 monclient: found mon.noname-a 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e80add00 con 0x7f19e806c380 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 10 monclient: _finish_auth 0 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 10 monclient: _check_auth_tickets 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f19d8001670 con 0x7f19e806c380 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19ecc25640 5 monclient: authenticate success, global_id 4524 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19ecc25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:21.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19e559a640 10 monclient: dump: 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:21.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.723+0000 7f19ecc25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19d8001890 con 0x7f19e806c380 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19e559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19e559a640 1 -- 172.21.15.69:0/4284519624 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f19d8001ba0 con 0x7f19e806c380 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19ecc25640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19ecc25640 4 set_mon_vals no callback set 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19e559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19e559a640 10 monclient: dump: 2024-03-25T19:47:21.739 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:21.740 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:21.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19ecc25640 10 monclient: shutdown 2024-03-25T19:47:21.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19ecc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:21.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.739+0000 7f19ecc25640 1 -- 172.21.15.69:0/4284519624 >> v1:172.21.15.69:6789/0 conn(0x7f19e806c380 legacy=0x7f19e80b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.742 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.740+0000 7f19ecc25640 1 -- 172.21.15.69:0/4284519624 shutdown_connections 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.740+0000 7f19ecc25640 1 -- 172.21.15.69:0/4284519624 >> 172.21.15.69:0/4284519624 conn(0x7f19e80b5c00 msgr2=0x7f19e80b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.740+0000 7f19ecc25640 1 -- 172.21.15.69:0/4284519624 shutdown_connections 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.740+0000 7f19ecc25640 1 -- 172.21.15.69:0/4284519624 wait complete. 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 10 monclient: build_initial_monmap 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 10 monclient: monmap: 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:21.743 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 1 librados: starting msgr at 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 1 librados: starting objecter 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 1 Processor -- start 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 1 -- start start 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 1 librados: setting wanted keys 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 1 librados: calling monclient init 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.741+0000 7f19ecc25640 10 monclient: init 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 10 monclient(hunting): picked mon.noname-c con 0x7f19e806c380 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 10 monclient(hunting): picked mon.noname-a con 0x7f19e806d670 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:21.744 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 10 monclient(hunting): picked mon.noname-b con 0x7f19e80af8f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:21.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e813f580 con 0x7f19e806d670 2024-03-25T19:47:21.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e813f7d0 con 0x7f19e80af8f0 2024-03-25T19:47:21.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e813fab0 con 0x7f19e806c380 2024-03-25T19:47:21.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:21.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.742+0000 7f19ecc25640 10 monclient(hunting): authenticate will time out at 5254.010880s 2024-03-25T19:47:21.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.754+0000 7f19e5d9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f19e806d670 0x7f19e8140220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57814/0 (socket says 172.21.15.69:57814) 2024-03-25T19:47:21.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.754+0000 7f19e5d9b640 1 -- 172.21.15.69:0/1404938277 learned_addr learned my addr 172.21.15.69:0/1404938277 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:21.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.757+0000 7f19d6ffd640 10 client.?.objecter ms_handle_connect 0x7f19e806c380 2024-03-25T19:47:21.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.757+0000 7f19d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.758+0000 7f19d6ffd640 10 client.?.objecter ms_handle_connect 0x7f19e806d670 2024-03-25T19:47:21.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.758+0000 7f19d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.760 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.760+0000 7f19d6ffd640 10 client.?.objecter ms_handle_connect 0x7f19e80af8f0 2024-03-25T19:47:21.760 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.760+0000 7f19d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.768+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918809429 0 0) 0x7f19e813f580 con 0x7f19e806d670 2024-03-25T19:47:21.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.769+0000 7f19d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.769+0000 7f19d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.769+0000 7f19d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.769+0000 7f19d6ffd640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:47:21.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.769+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c8004eb0 con 0x7f19e806d670 2024-03-25T19:47:21.770 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.770+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093184212 0 0) 0x7f19e813fab0 con 0x7f19e806c380 2024-03-25T19:47:21.770 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.770+0000 7f19d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.770 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.770+0000 7f19d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.770 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.770+0000 7f19d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.770 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.770+0000 7f19d6ffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:47:21.771 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.770+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e813f580 con 0x7f19e806c380 2024-03-25T19:47:21.771 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.771+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213866952 0 0) 0x7f19e813f7d0 con 0x7f19e80af8f0 2024-03-25T19:47:21.771 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.771+0000 7f19d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.771 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.771+0000 7f19d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.772 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.771+0000 7f19d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.772 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.771+0000 7f19d6ffd640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:47:21.772 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.771+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e813fab0 con 0x7f19e80af8f0 2024-03-25T19:47:21.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.780+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2561850789 0 0) 0x7f19e813fab0 con 0x7f19e80af8f0 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.780+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19e813f7d0 con 0x7f19e80af8f0 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.780+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f19e0001bd0 con 0x7f19e80af8f0 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.780+0000 7f19d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:21.782 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928999824 0 0) 0x7f19c8004eb0 con 0x7f19e806d670 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19e813fab0 con 0x7f19e806d670 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f19dc001910 con 0x7f19e806d670 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.781+0000 7f19d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:21.783 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:21.784 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:21.784 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:21.784 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.784 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:21.784 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:21.784 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:21.785 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:21.785 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:21.790 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2987563288 0 0) 0x7f19e813fab0 con 0x7f19e806d670 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 >> v1:172.21.15.69:6790/0 conn(0x7f19e806c380 legacy=0x7f19e8148300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 >> v1:172.21.15.169:6789/0 conn(0x7f19e80af8f0 legacy=0x7f19e81e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 1 monclient: found mon.a 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e813fd90 con 0x7f19e806d670 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 5 monclient: authenticate success, global_id 4527 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 10 monclient: _renew_subs 2024-03-25T19:47:21.791 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:21.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e813eba0 con 0x7f19e806d670 2024-03-25T19:47:21.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:21.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 10 monclient: _renew_subs 2024-03-25T19:47:21.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:21.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e8144d20 con 0x7f19e806d670 2024-03-25T19:47:21.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.790+0000 7f19ecc25640 1 librados: init done 2024-03-25T19:47:21.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.791+0000 7f19ecc25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:21.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.791+0000 7f19ecc25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:21.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.791+0000 7f19ecc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:21.793 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:21.791+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e80af480 con 0x7f19e806d670 2024-03-25T19:47:22.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.089+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19dc0047e0 con 0x7f19e806d670 2024-03-25T19:47:22.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.090+0000 7f19d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:22.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.090+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f19dc001d80 con 0x7f19e806d670 2024-03-25T19:47:22.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.090+0000 7f19d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:22.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.090+0000 7f19d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:22.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.090+0000 7f19d6ffd640 10 monclient: dump: 2024-03-25T19:47:22.092 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:22.092 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:22.092 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:22.092 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:22.093 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:22.093 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:22.093 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:22.093 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:22.093 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:22.093 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:22.093 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.090+0000 7f19e4d99640 4 set_mon_vals no callback set 2024-03-25T19:47:22.093 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.090+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f19dc004c30 con 0x7f19e806d670 2024-03-25T19:47:22.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.091+0000 7f19e659c640 10 monclient: get_auth_request con 0x7f19c806dfb0 auth_method 0 2024-03-25T19:47:22.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.091+0000 7f19d6ffd640 10 client.4527.objecter ms_handle_connect 0x7f19c806dfb0 2024-03-25T19:47:22.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.096+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f19dc0547e0 con 0x7f19e806d670 2024-03-25T19:47:22.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.096+0000 7f19d6ffd640 10 client.4527.objecter ms_dispatch 0x7f19e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:22.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.096+0000 7f19d6ffd640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:22.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.096+0000 7f19d6ffd640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:22.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.096+0000 7f19d6ffd640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T19:47:22.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.096+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19dc050120 con 0x7f19e806d670 2024-03-25T19:47:22.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.096+0000 7f19d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:22.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.096+0000 7f19d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:22.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.348+0000 7f19ecc25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:22.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.348+0000 7f19ecc25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:22.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.348+0000 7f19ecc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:22.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.348+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19e80627b0 con 0x7f19e806d670 2024-03-25T19:47:22.401 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.401+0000 7f19d6ffd640 1 -- 172.21.15.69:0/1404938277 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f19dc052220 con 0x7f19e806d670 2024-03-25T19:47:22.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.401+0000 7f19d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:22.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.401+0000 7f19d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:22.402 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:47:22.402 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 10 librados: watch_flush enter 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 10 librados: watch_flush exit 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 10 client.4527.objecter successfully canceled tick 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 >> v1:172.21.15.169:6800/3072548259 conn(0x7f19c806dfb0 legacy=0x7f19c8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 10 monclient: shutdown 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 >> v1:172.21.15.69:6789/0 conn(0x7f19e806d670 legacy=0x7f19e8140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 shutdown_connections 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 >> 172.21.15.69:0/1404938277 conn(0x7f19e80b5c00 msgr2=0x7f19e8070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 shutdown_connections 2024-03-25T19:47:22.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 1 -- 172.21.15.69:0/1404938277 wait complete. 2024-03-25T19:47:22.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:22.403+0000 7f19ecc25640 1 librados: shutdown 2024-03-25T19:47:22.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:28.060 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:06.023 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:28.107 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:06.071 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:28.142 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:06.106 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:28.151 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:06.105 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:28.156 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:06.110 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:28.172 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:06.136 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:28.177 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:06.141 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:28.187 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:06.141 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:28.207 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:06.171 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:29.223 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.323+0000 7fb3ae2c5640 10 monclient: get_monmap_and_config 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.323+0000 7fb3ae2c5640 10 monclient: build_initial_monmap 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.323+0000 7fb3ae2c5640 10 monclient: monmap: 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:47:29.325123+0000 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:47:29.325123+0000 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:29.324 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:29.325 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:47:29.325 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:47:29.325 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:29.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.324+0000 7fb3ae2c5640 1 Processor -- start 2024-03-25T19:47:29.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.324+0000 7fb3ae2c5640 1 -- start start 2024-03-25T19:47:29.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.324+0000 7fb3ae2c5640 10 monclient: init 2024-03-25T19:47:29.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:29.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:29.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a80a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:29.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:29.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a81430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:29.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a8142520 con 0x7fb3a813f710 2024-03-25T19:47:29.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a80b28a0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a80b2a30 con 0x7fb3a81430c0 2024-03-25T19:47:29.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:29.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.325+0000 7fb3ae2c5640 10 monclient(hunting): authenticate will time out at 5261.594424s 2024-03-25T19:47:29.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.326+0000 7fb3a7fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb3a80a98f0 0x7fb3a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:36340/0 (socket says 172.21.15.69:36340) 2024-03-25T19:47:29.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.326+0000 7fb3a7fff640 1 -- 172.21.15.69:0/1812755600 learned_addr learned my addr 172.21.15.69:0/1812755600 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:29.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.340+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443121724 0 0) 0x7fb3a80b2a30 con 0x7fb3a81430c0 2024-03-25T19:47:29.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.340+0000 7fb3a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.340+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.340+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.340+0000 7fb3a6ffd640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:47:29.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.340+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb390003520 con 0x7fb3a81430c0 2024-03-25T19:47:29.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.347+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867337444 0 0) 0x7fb3a8142520 con 0x7fb3a813f710 2024-03-25T19:47:29.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.347+0000 7fb3a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.347+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.347+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.347+0000 7fb3a6ffd640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:47:29.347 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.347+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a80b2a30 con 0x7fb3a813f710 2024-03-25T19:47:29.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.348+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623638012 0 0) 0x7fb3a80b28a0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.348+0000 7fb3a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.348+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.348+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.348+0000 7fb3a6ffd640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:47:29.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.348+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a8142520 con 0x7fb3a80a98f0 2024-03-25T19:47:29.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956525418 0 0) 0x7fb3a8142520 con 0x7fb3a80a98f0 2024-03-25T19:47:29.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 >> v1:172.21.15.69:6790/0 conn(0x7fb3a81430c0 legacy=0x7fb3a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 >> v1:172.21.15.69:6789/0 conn(0x7fb3a813f710 legacy=0x7fb3a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a80b44c0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb3980015b0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3a6ffd640 10 monclient: dump: 2024-03-25T19:47:29.354 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3ae2c5640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:47:29.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.353+0000 7fb3ae2c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:29.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.362+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb398001710 con 0x7fb3a80a98f0 2024-03-25T19:47:29.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.362+0000 7fb3a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:29.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.362+0000 7fb3a6ffd640 1 -- 172.21.15.69:0/1812755600 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb398001b20 con 0x7fb3a80a98f0 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.362+0000 7fb3ae2c5640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.362+0000 7fb3ae2c5640 4 set_mon_vals no callback set 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.362+0000 7fb3a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.362+0000 7fb3a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.362+0000 7fb3a6ffd640 10 monclient: dump: 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.363+0000 7fb3ae2c5640 10 monclient: shutdown 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.363+0000 7fb3ae2c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.363+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1812755600 >> v1:172.21.15.169:6789/0 conn(0x7fb3a80a98f0 legacy=0x7fb3a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.363+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1812755600 shutdown_connections 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.363+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1812755600 >> 172.21.15.69:0/1812755600 conn(0x7fb3a80b7ea0 msgr2=0x7fb3a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.363+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1812755600 shutdown_connections 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.363+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1812755600 wait complete. 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 10 monclient: build_initial_monmap 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 10 monclient: monmap: 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:29.366 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 1 librados: starting msgr at 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 1 librados: starting objecter 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 1 Processor -- start 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 1 -- start start 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 1 librados: setting wanted keys 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 1 librados: calling monclient init 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.364+0000 7fb3ae2c5640 10 monclient: init 2024-03-25T19:47:29.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a80a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a81d41e0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a81d44c0 con 0x7fb3a81430c0 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a8138b10 con 0x7fb3a813f710 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.365+0000 7fb3ae2c5640 10 monclient(hunting): authenticate will time out at 5261.634147s 2024-03-25T19:47:29.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.368+0000 7fb3ac862640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb3a81430c0 0x7fb3a81e2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:36368/0 (socket says 172.21.15.69:36368) 2024-03-25T19:47:29.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.368+0000 7fb3ac862640 1 -- 172.21.15.69:0/1857081463 learned_addr learned my addr 172.21.15.69:0/1857081463 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:29.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.371+0000 7fb3a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb3a81430c0 2024-03-25T19:47:29.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.371+0000 7fb3a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.373+0000 7fb3a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb3a80a98f0 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.373+0000 7fb3a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:29.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.375+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103147821 0 0) 0x7fb3a81d44c0 con 0x7fb3a81430c0 2024-03-25T19:47:29.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.375+0000 7fb3a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.375+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.375+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.375+0000 7fb3a4ff9640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:47:29.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.375+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb394004eb0 con 0x7fb3a81430c0 2024-03-25T19:47:29.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.380+0000 7fb3a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb3a813f710 2024-03-25T19:47:29.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.380+0000 7fb3a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:29.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.391+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659041836 0 0) 0x7fb3a81d41e0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.391+0000 7fb3a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.391+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.391+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.391+0000 7fb3a4ff9640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:47:29.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.391+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a81d44c0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.392+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475063636 0 0) 0x7fb3a8138b10 con 0x7fb3a813f710 2024-03-25T19:47:29.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.392+0000 7fb3a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.392+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.392+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.392+0000 7fb3a4ff9640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:47:29.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.392+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a81d41e0 con 0x7fb3a813f710 2024-03-25T19:47:29.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.396+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 854618322 0 0) 0x7fb394004eb0 con 0x7fb3a81430c0 2024-03-25T19:47:29.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.396+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3a8138b10 con 0x7fb3a81430c0 2024-03-25T19:47:29.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.396+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb3a0001c30 con 0x7fb3a81430c0 2024-03-25T19:47:29.396 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.396+0000 7fb3a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.396+0000 7fb3a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.396+0000 7fb3a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:29.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.396+0000 7fb3a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.408+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534622015 0 0) 0x7fb3a81d44c0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.409+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb394004eb0 con 0x7fb3a80a98f0 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.409+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb398001a20 con 0x7fb3a80a98f0 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.409+0000 7fb3a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.409+0000 7fb3a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.409+0000 7fb3a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.409 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:29.410 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:29.410 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:29.410 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:29.410 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:29.410 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2334561736 0 0) 0x7fb3a8138b10 con 0x7fb3a81430c0 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 >> v1:172.21.15.69:6790/0 conn(0x7fb3a813f710 legacy=0x7fb3a81d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 >> v1:172.21.15.69:6789/0 conn(0x7fb3a80a98f0 legacy=0x7fb3a8138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 1 monclient: found mon.b 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8135220 con 0x7fb3a81430c0 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:29.416 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 10 monclient: _renew_subs 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a81d5060 con 0x7fb3a81430c0 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 10 monclient: _renew_subs 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a8138f20 con 0x7fb3a81430c0 2024-03-25T19:47:29.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.415+0000 7fb3ae2c5640 1 librados: init done 2024-03-25T19:47:29.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.417+0000 7fb3ae2c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:29.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.417+0000 7fb3ae2c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:29.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.417+0000 7fb3ae2c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:29.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.417+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a80b3ef0 con 0x7fb3a81430c0 2024-03-25T19:47:29.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.705+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3a00020e0 con 0x7fb3a81430c0 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.705+0000 7fb3a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.705+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb3a0003b30 con 0x7fb3a81430c0 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.705+0000 7fb3a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.705+0000 7fb3a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.705+0000 7fb3a4ff9640 10 monclient: dump: 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:29.706 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:29.707 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:29.707 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:29.707 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:29.707 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:29.707 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.705+0000 7fb3a67fc640 4 set_mon_vals no callback set 2024-03-25T19:47:29.707 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.706+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fb3a0003d50 con 0x7fb3a81430c0 2024-03-25T19:47:29.707 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.706+0000 7fb3a7fff640 10 monclient: get_auth_request con 0x7fb39406df20 auth_method 0 2024-03-25T19:47:29.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.707+0000 7fb3a4ff9640 10 client.4501.objecter ms_handle_connect 0x7fb39406df20 2024-03-25T19:47:29.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.712+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fb3a0054ad0 con 0x7fb3a81430c0 2024-03-25T19:47:29.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.712+0000 7fb3a4ff9640 10 client.4501.objecter ms_dispatch 0x7fb3a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:29.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.712+0000 7fb3a4ff9640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:29.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.712+0000 7fb3a4ff9640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:29.712 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.712+0000 7fb3a4ff9640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T19:47:29.713 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.712+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3a00505e0 con 0x7fb3a81430c0 2024-03-25T19:47:29.713 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.712+0000 7fb3a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:29.713 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.712+0000 7fb3a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:29.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.966+0000 7fb3ae2c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.966+0000 7fb3ae2c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.966+0000 7fb3ae2c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:29.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.966+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3a80ac020 con 0x7fb3a81430c0 2024-03-25T19:47:29.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.987+0000 7fb3a4ff9640 1 -- 172.21.15.69:0/1857081463 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fb3a80ac020 con 0x7fb3a81430c0 2024-03-25T19:47:29.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.987+0000 7fb3a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.987+0000 7fb3a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:29.987 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:47:29.988 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:29.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 10 librados: watch_flush enter 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 10 librados: watch_flush exit 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 10 client.4501.objecter successfully canceled tick 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 >> v1:172.21.15.169:6800/3072548259 conn(0x7fb39406df20 legacy=0x7fb3940703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 10 monclient: shutdown 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 >> v1:172.21.15.169:6789/0 conn(0x7fb3a81430c0 legacy=0x7fb3a81e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 shutdown_connections 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 >> 172.21.15.69:0/1857081463 conn(0x7fb3a80b7ea0 msgr2=0x7fb3a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 shutdown_connections 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 1 -- 172.21.15.69:0/1857081463 wait complete. 2024-03-25T19:47:29.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:29.989+0000 7fb3ae2c5640 1 librados: shutdown 2024-03-25T19:47:30.001 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:36.807 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:36.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.921+0000 7efc47115640 10 monclient: get_monmap_and_config 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.921+0000 7efc47115640 10 monclient: build_initial_monmap 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.921+0000 7efc47115640 10 monclient: monmap: 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:47:36.922897+0000 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:47:36.922897+0000 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:47:36.922 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.922+0000 7efc47115640 1 Processor -- start 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.922+0000 7efc47115640 1 -- start start 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.922+0000 7efc47115640 10 monclient: init 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 10 monclient(hunting): picked mon.noname-a con 0x7efc400b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 10 monclient(hunting): picked mon.noname-b con 0x7efc400b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 10 monclient(hunting): picked mon.noname-c con 0x7efc4014be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc400af940 con 0x7efc400b4d60 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc400b0a70 con 0x7efc4014be90 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc400b0d50 con 0x7efc400b1730 2024-03-25T19:47:36.924 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:36.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.923+0000 7efc47115640 10 monclient(hunting): authenticate will time out at 5269.192222s 2024-03-25T19:47:36.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.927+0000 7efc456b2640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7efc4014be90 0x7efc4014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:46812/0 (socket says 172.21.15.69:46812) 2024-03-25T19:47:36.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.927+0000 7efc456b2640 1 -- 172.21.15.69:0/3079728031 learned_addr learned my addr 172.21.15.69:0/3079728031 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:36.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.937+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289946856 0 0) 0x7efc400b0a70 con 0x7efc4014be90 2024-03-25T19:47:36.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.937+0000 7efc3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.937+0000 7efc3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.937+0000 7efc3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.937+0000 7efc3f7fe640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:47:36.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.937+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc30003520 con 0x7efc4014be90 2024-03-25T19:47:36.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.942+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617298590 0 0) 0x7efc400b0d50 con 0x7efc400b1730 2024-03-25T19:47:36.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.942+0000 7efc3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.942+0000 7efc3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.942+0000 7efc3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.942+0000 7efc3f7fe640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:47:36.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.942+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc400b0a70 con 0x7efc400b1730 2024-03-25T19:47:36.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.954+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970746653 0 0) 0x7efc400af940 con 0x7efc400b4d60 2024-03-25T19:47:36.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.954+0000 7efc3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.954+0000 7efc3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.954+0000 7efc3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.954+0000 7efc3f7fe640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:47:36.955 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.954+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc400b0d50 con 0x7efc400b4d60 2024-03-25T19:47:36.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 256571798 0 0) 0x7efc30003520 con 0x7efc4014be90 2024-03-25T19:47:36.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 >> v1:172.21.15.69:6790/0 conn(0x7efc400b1730 legacy=0x7efc400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 >> v1:172.21.15.69:6789/0 conn(0x7efc400b4d60 legacy=0x7efc4013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:36.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:47:36.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc400b1030 con 0x7efc4014be90 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7efc38001a10 con 0x7efc4014be90 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc3f7fe640 10 monclient: dump: 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:36.963 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.964 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:36.964 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:36.964 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:36.964 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:36.964 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:36.964 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc47115640 5 monclient: authenticate success, global_id 4504 2024-03-25T19:47:36.964 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.962+0000 7efc47115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:36.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.976+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc38001b70 con 0x7efc4014be90 2024-03-25T19:47:36.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.976+0000 7efc3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.976+0000 7efc3f7fe640 1 -- 172.21.15.69:0/3079728031 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7efc38001f80 con 0x7efc4014be90 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.976+0000 7efc3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.976+0000 7efc3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.976+0000 7efc3f7fe640 10 monclient: dump: 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:36.977 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:36.978 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:36.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.976+0000 7efc47115640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:36.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.976+0000 7efc47115640 4 set_mon_vals no callback set 2024-03-25T19:47:36.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.977+0000 7efc47115640 10 monclient: shutdown 2024-03-25T19:47:36.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.977+0000 7efc47115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:36.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.977+0000 7efc47115640 1 -- 172.21.15.69:0/3079728031 >> v1:172.21.15.169:6789/0 conn(0x7efc4014be90 legacy=0x7efc4014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.977+0000 7efc47115640 1 -- 172.21.15.69:0/3079728031 shutdown_connections 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.977+0000 7efc47115640 1 -- 172.21.15.69:0/3079728031 >> 172.21.15.69:0/3079728031 conn(0x7efc400617a0 msgr2=0x7efc40061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.977+0000 7efc47115640 1 -- 172.21.15.69:0/3079728031 shutdown_connections 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.977+0000 7efc47115640 1 -- 172.21.15.69:0/3079728031 wait complete. 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.978+0000 7efc47115640 10 monclient: build_initial_monmap 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.978+0000 7efc47115640 10 monclient: monmap: 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:47:36.980 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:47:36.981 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:36.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.979+0000 7efc47115640 1 librados: starting msgr at 2024-03-25T19:47:36.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.979+0000 7efc47115640 1 librados: starting objecter 2024-03-25T19:47:36.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.979+0000 7efc47115640 1 Processor -- start 2024-03-25T19:47:36.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.979+0000 7efc47115640 1 -- start start 2024-03-25T19:47:36.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.979+0000 7efc47115640 1 librados: setting wanted keys 2024-03-25T19:47:36.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.979+0000 7efc47115640 1 librados: calling monclient init 2024-03-25T19:47:36.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.979+0000 7efc47115640 10 monclient: init 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 10 monclient(hunting): picked mon.noname-a con 0x7efc400b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 10 monclient(hunting): picked mon.noname-c con 0x7efc400b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 10 monclient(hunting): picked mon.noname-b con 0x7efc4014be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc401363f0 con 0x7efc400b1730 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc401374e0 con 0x7efc4014be90 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc40138140 con 0x7efc400b4d60 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:36.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.980+0000 7efc47115640 10 monclient(hunting): authenticate will time out at 5269.249104s 2024-03-25T19:47:36.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.985+0000 7efc3ffff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7efc400b4d60 0x7efc4012fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:38246/0 (socket says 172.21.15.69:38246) 2024-03-25T19:47:36.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.985+0000 7efc3ffff640 1 -- 172.21.15.69:0/1295358593 learned_addr learned my addr 172.21.15.69:0/1295358593 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:36.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.986+0000 7efc3d7fa640 10 client.?.objecter ms_handle_connect 0x7efc400b4d60 2024-03-25T19:47:36.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.986+0000 7efc3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:36.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.997+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174962150 0 0) 0x7efc40138140 con 0x7efc400b4d60 2024-03-25T19:47:36.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.997+0000 7efc3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.997 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.997+0000 7efc3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.997+0000 7efc3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.997+0000 7efc3d7fa640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:47:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:36.997+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc24004eb0 con 0x7efc400b4d60 2024-03-25T19:47:37.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.000+0000 7efc3d7fa640 10 client.?.objecter ms_handle_connect 0x7efc400b1730 2024-03-25T19:47:37.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.000+0000 7efc3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:37.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.004+0000 7efc3d7fa640 10 client.?.objecter ms_handle_connect 0x7efc4014be90 2024-03-25T19:47:37.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.004+0000 7efc3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.006+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2477233686 0 0) 0x7efc24004eb0 con 0x7efc400b4d60 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.006+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc40138140 con 0x7efc400b4d60 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.006+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7efc2c001a30 con 0x7efc400b4d60 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.006+0000 7efc3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.006+0000 7efc3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.006+0000 7efc3d7fa640 10 monclient(hunting): dump: 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:37.006 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:37.007 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:37.007 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:37.007 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:37.007 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:37.007 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:37.007 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:37.007 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:37.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.006+0000 7efc3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2715373030 0 0) 0x7efc40138140 con 0x7efc400b4d60 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 >> v1:172.21.15.169:6789/0 conn(0x7efc4014be90 legacy=0x7efc40143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 >> v1:172.21.15.69:6789/0 conn(0x7efc400b1730 legacy=0x7efc40136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 1 monclient: found mon.c 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc40139090 con 0x7efc400b4d60 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:47:37.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 5 monclient: authenticate success, global_id 4520 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 10 monclient: _renew_subs 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc40136030 con 0x7efc400b4d60 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 10 monclient: _renew_subs 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc40130650 con 0x7efc400b4d60 2024-03-25T19:47:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.012+0000 7efc47115640 1 librados: init done 2024-03-25T19:47:37.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.013+0000 7efc47115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:37.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.013+0000 7efc47115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:37.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.013+0000 7efc47115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:37.015 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.013+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc400ae6c0 con 0x7efc400b4d60 2024-03-25T19:47:37.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.313+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc2c001cd0 con 0x7efc400b4d60 2024-03-25T19:47:37.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.313+0000 7efc3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:37.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.313+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7efc2c003a60 con 0x7efc400b4d60 2024-03-25T19:47:37.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.313+0000 7efc3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:37.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.313+0000 7efc3d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:37.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.313+0000 7efc3d7fa640 10 monclient: dump: 2024-03-25T19:47:37.314 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:37.314 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:37.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.313+0000 7efc3effd640 4 set_mon_vals no callback set 2024-03-25T19:47:37.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.314+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7efc2c003c80 con 0x7efc400b4d60 2024-03-25T19:47:37.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.315+0000 7efc44eb1640 10 monclient: get_auth_request con 0x7efc2406dc70 auth_method 0 2024-03-25T19:47:37.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.315+0000 7efc3d7fa640 10 client.4520.objecter ms_handle_connect 0x7efc2406dc70 2024-03-25T19:47:37.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.322+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7efc2c0548b0 con 0x7efc400b4d60 2024-03-25T19:47:37.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.322+0000 7efc3d7fa640 10 client.4520.objecter ms_dispatch 0x7efc40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:37.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.322+0000 7efc3d7fa640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:37.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.322+0000 7efc3d7fa640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:37.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.322+0000 7efc3d7fa640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-25T19:47:37.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.322+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc2c0503c0 con 0x7efc400b4d60 2024-03-25T19:47:37.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.322+0000 7efc3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:37.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.322+0000 7efc3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:37.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.580+0000 7efc47115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:37.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.580+0000 7efc47115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:37.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.580+0000 7efc47115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:47:37.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.580+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc400627b0 con 0x7efc400b4d60 2024-03-25T19:47:37.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.611+0000 7efc3d7fa640 1 -- 172.21.15.69:0/1295358593 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7efc2c0522f0 con 0x7efc400b4d60 2024-03-25T19:47:37.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.611+0000 7efc3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:37.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.611+0000 7efc3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:37.612 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:47:37.612 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 10 librados: watch_flush enter 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 10 librados: watch_flush exit 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 10 client.4520.objecter successfully canceled tick 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 >> v1:172.21.15.169:6800/3072548259 conn(0x7efc2406dc70 legacy=0x7efc24070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 10 monclient: shutdown 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 >> v1:172.21.15.69:6790/0 conn(0x7efc400b4d60 legacy=0x7efc4012fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 shutdown_connections 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 >> 172.21.15.69:0/1295358593 conn(0x7efc400617a0 msgr2=0x7efc4013c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 shutdown_connections 2024-03-25T19:47:37.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 1 -- 172.21.15.69:0/1295358593 wait complete. 2024-03-25T19:47:37.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:37.614+0000 7efc47115640 1 librados: shutdown 2024-03-25T19:47:37.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:44.431 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:44.551 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.552+0000 7f6da096d640 10 monclient: get_monmap_and_config 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.552+0000 7f6da096d640 10 monclient: build_initial_monmap 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.552+0000 7f6da096d640 10 monclient: monmap: 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:47:44.552914+0000 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:47:44.552914+0000 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:47:44.552 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.553+0000 7f6da096d640 1 Processor -- start 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.553+0000 7f6da096d640 1 -- start start 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.553+0000 7f6da096d640 10 monclient: init 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c13f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c142520 con 0x7f6d9c13f710 2024-03-25T19:47:44.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0b28a0 con 0x7f6d9c1430c0 2024-03-25T19:47:44.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0b2a30 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:44.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6da096d640 10 monclient(hunting): authenticate will time out at 5276.822065s 2024-03-25T19:47:44.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6d9a59c640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f6d9c0a98f0 0x7f6d9c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:39170/0 (socket says 172.21.15.69:39170) 2024-03-25T19:47:44.555 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.554+0000 7f6d9a59c640 1 -- 172.21.15.69:0/694159412 learned_addr learned my addr 172.21.15.69:0/694159412 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:44.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.629+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022977640 0 0) 0x7f6d9c142520 con 0x7f6d9c13f710 2024-03-25T19:47:44.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.630+0000 7f6d99d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.630+0000 7f6d99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.630+0000 7f6d99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.630+0000 7f6d99d9b640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:47:44.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.630+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d88003520 con 0x7f6d9c13f710 2024-03-25T19:47:44.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1436564184 0 0) 0x7f6d88003520 con 0x7f6d9c13f710 2024-03-25T19:47:44.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 >> v1:172.21.15.69:6790/0 conn(0x7f6d9c0a98f0 legacy=0x7f6d9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 >> v1:172.21.15.169:6789/0 conn(0x7f6d9c1430c0 legacy=0x7f6d9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 1 monclient: found mon.noname-a 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0b44c0 con 0x7f6d9c13f710 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 10 monclient: _finish_auth 0 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6d84001690 con 0x7f6d9c13f710 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6d99d9b640 10 monclient: dump: 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:44.647 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6da096d640 5 monclient: authenticate success, global_id 4542 2024-03-25T19:47:44.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.647+0000 7f6da096d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d840017f0 con 0x7f6d9c13f710 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6d99d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6d99d9b640 1 -- 172.21.15.69:0/694159412 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6d84001c00 con 0x7f6d9c13f710 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 4 set_mon_vals no callback set 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6d99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6d99d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6d99d9b640 10 monclient: dump: 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:44.656 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:44.657 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:44.657 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:44.657 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.657 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:44.657 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:44.657 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:44.657 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:44.657 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:44.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 10 monclient: shutdown 2024-03-25T19:47:44.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:44.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 1 -- 172.21.15.69:0/694159412 >> v1:172.21.15.69:6789/0 conn(0x7f6d9c13f710 legacy=0x7f6d9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 1 -- 172.21.15.69:0/694159412 shutdown_connections 2024-03-25T19:47:44.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 1 -- 172.21.15.69:0/694159412 >> 172.21.15.69:0/694159412 conn(0x7f6d9c0b7ea0 msgr2=0x7f6d9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:44.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 1 -- 172.21.15.69:0/694159412 shutdown_connections 2024-03-25T19:47:44.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.657+0000 7f6da096d640 1 -- 172.21.15.69:0/694159412 wait complete. 2024-03-25T19:47:44.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.658+0000 7f6da096d640 10 monclient: build_initial_monmap 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.658+0000 7f6da096d640 10 monclient: monmap: 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 librados: starting msgr at 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 librados: starting objecter 2024-03-25T19:47:44.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 Processor -- start 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 -- start start 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 librados: setting wanted keys 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 librados: calling monclient init 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 10 monclient: init 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c13f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c138980 con 0x7f6d9c1430c0 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c138640 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0b1700 con 0x7f6d9c13f710 2024-03-25T19:47:44.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:44.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.659+0000 7f6da096d640 10 monclient(hunting): authenticate will time out at 5276.927381s 2024-03-25T19:47:44.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.676+0000 7f6d9ad9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6d9c1430c0 0x7f6d9c1db000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41024/0 (socket says 172.21.15.69:41024) 2024-03-25T19:47:44.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.676+0000 7f6d9ad9d640 1 -- 172.21.15.69:0/3557508691 learned_addr learned my addr 172.21.15.69:0/3557508691 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:44.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.679+0000 7f6d92ffd640 10 client.?.objecter ms_handle_connect 0x7f6d9c1430c0 2024-03-25T19:47:44.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.680+0000 7f6d92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.692+0000 7f6d92ffd640 10 client.?.objecter ms_handle_connect 0x7f6d9c0a98f0 2024-03-25T19:47:44.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.692+0000 7f6d92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.696+0000 7f6d92ffd640 10 client.?.objecter ms_handle_connect 0x7f6d9c13f710 2024-03-25T19:47:44.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.696+0000 7f6d92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.703+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650195544 0 0) 0x7f6d9c138640 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.703+0000 7f6d92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.703+0000 7f6d92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.703+0000 7f6d92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.703+0000 7f6d92ffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:47:44.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.703+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c004eb0 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.709+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908584785 0 0) 0x7f6d9c0b1700 con 0x7f6d9c13f710 2024-03-25T19:47:44.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.709+0000 7f6d92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.709+0000 7f6d92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.709+0000 7f6d92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.709+0000 7f6d92ffd640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:47:44.708 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.709+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c138640 con 0x7f6d9c13f710 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.719+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3580437259 0 0) 0x7f6d7c004eb0 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.719+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9c0b1700 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.719+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6d8c0018a0 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.719+0000 7f6d92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.719+0000 7f6d92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.719+0000 7f6d92ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.719 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:44.720 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:44.720 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:44.720 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:44.720 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:44.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.720+0000 7f6d92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:44.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 582336687 0 0) 0x7f6d9c0b1700 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 >> v1:172.21.15.69:6790/0 conn(0x7f6d9c13f710 legacy=0x7f6d9c1da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 >> v1:172.21.15.69:6789/0 conn(0x7f6d9c1430c0 legacy=0x7f6d9c1db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:44.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 1 monclient: found mon.b 2024-03-25T19:47:44.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:44.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:44.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0b1290 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6d92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 10 monclient: _renew_subs 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c1e2eb0 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 10 monclient: _renew_subs 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c1e3220 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.729+0000 7f6da096d640 1 librados: init done 2024-03-25T19:47:44.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.730+0000 7f6da096d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:44.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.730+0000 7f6da096d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:44.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.730+0000 7f6da096d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:44.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.730+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c0b14e0 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.736+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d8c001d10 con 0x7f6d9c0a98f0 2024-03-25T19:47:44.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.736+0000 7f6d92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:44.735 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:44.736+0000 7f6d9959a640 4 set_mon_vals no callback set 2024-03-25T19:47:45.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.016+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6d8c0048a0 con 0x7f6d9c0a98f0 2024-03-25T19:47:45.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.017+0000 7f6d92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:45.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.017+0000 7f6d92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:45.016 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.017+0000 7f6d92ffd640 10 monclient: dump: 2024-03-25T19:47:45.016 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:45.016 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:45.016 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:45.016 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:45.017 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:45.017 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:45.017 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:45.017 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:45.017 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:45.017 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:45.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.017+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f6d8c004a80 con 0x7f6d9c0a98f0 2024-03-25T19:47:45.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.018+0000 7f6d93fff640 10 monclient: get_auth_request con 0x7f6d7c06dcc0 auth_method 0 2024-03-25T19:47:45.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.018+0000 7f6d92ffd640 10 client.4513.objecter ms_handle_connect 0x7f6d7c06dcc0 2024-03-25T19:47:45.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.022+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f6d8c054610 con 0x7f6d9c0a98f0 2024-03-25T19:47:45.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.022+0000 7f6d92ffd640 10 client.4513.objecter ms_dispatch 0x7f6d9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.022+0000 7f6d92ffd640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.022+0000 7f6d92ffd640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:45.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.022+0000 7f6d92ffd640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T19:47:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.025+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d8c04ff50 con 0x7f6d9c0a98f0 2024-03-25T19:47:45.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.025+0000 7f6d92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:45.025 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.026+0000 7f6d92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:45.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.283+0000 7f6da096d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:45.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.283+0000 7f6da096d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:45.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.283+0000 7f6da096d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:45.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.283+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d9c0abb30 con 0x7f6d9c0a98f0 2024-03-25T19:47:45.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.307+0000 7f6d92ffd640 1 -- 172.21.15.69:0/3557508691 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f6d8c052050 con 0x7f6d9c0a98f0 2024-03-25T19:47:45.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.307+0000 7f6d92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:45.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.307+0000 7f6d92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:45.307 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:47:45.307 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:45.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.309+0000 7f6da096d640 10 librados: watch_flush enter 2024-03-25T19:47:45.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.309+0000 7f6da096d640 10 librados: watch_flush exit 2024-03-25T19:47:45.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.309+0000 7f6da096d640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T19:47:45.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.309+0000 7f6da096d640 10 client.4513.objecter successfully canceled tick 2024-03-25T19:47:45.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 >> v1:172.21.15.169:6800/3072548259 conn(0x7f6d7c06dcc0 legacy=0x7f6d7c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:45.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 10 monclient: shutdown 2024-03-25T19:47:45.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:45.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 >> v1:172.21.15.169:6789/0 conn(0x7f6d9c0a98f0 legacy=0x7f6d9c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:45.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 shutdown_connections 2024-03-25T19:47:45.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 >> 172.21.15.69:0/3557508691 conn(0x7f6d9c0b7ea0 msgr2=0x7f6d9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:45.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 shutdown_connections 2024-03-25T19:47:45.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 1 -- 172.21.15.69:0/3557508691 wait complete. 2024-03-25T19:47:45.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:45.310+0000 7f6da096d640 1 librados: shutdown 2024-03-25T19:47:45.321 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:48.175 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:26.139 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:48.183 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:26.147 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:48.213 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:26.177 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:48.234 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:26.198 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:48.238 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:26.202 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:48.260 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:26.214 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:48.265 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:26.219 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:48.269 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:26.233 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:48.293 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:26.247 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:52.128 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:52.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.241+0000 7fc792713640 10 monclient: get_monmap_and_config 2024-03-25T19:47:52.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.241+0000 7fc792713640 10 monclient: build_initial_monmap 2024-03-25T19:47:52.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.241+0000 7fc792713640 10 monclient: monmap: 2024-03-25T19:47:52.241 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:52.241 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:52.241 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:47:52.242427+0000 2024-03-25T19:47:52.241 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:47:52.242427+0000 2024-03-25T19:47:52.241 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:52.242 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:52.242 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:52.242 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:47:52.242 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:47:52.242 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 1 Processor -- start 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 1 -- start start 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 10 monclient: init 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 10 monclient(hunting): picked mon.noname-c con 0x7fc78c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 10 monclient(hunting): picked mon.noname-b con 0x7fc78c13f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 10 monclient(hunting): picked mon.noname-a con 0x7fc78c1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c142520 con 0x7fc78c1430c0 2024-03-25T19:47:52.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c0b28a0 con 0x7fc78c0a98f0 2024-03-25T19:47:52.244 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c0b2a30 con 0x7fc78c13f710 2024-03-25T19:47:52.244 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:52.244 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.242+0000 7fc792713640 10 monclient(hunting): authenticate will time out at 5284.511423s 2024-03-25T19:47:52.256 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.256+0000 7fc78bfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc78c0a98f0 0x7fc78c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:40778/0 (socket says 172.21.15.69:40778) 2024-03-25T19:47:52.256 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.256+0000 7fc78bfff640 1 -- 172.21.15.69:0/280014790 learned_addr learned my addr 172.21.15.69:0/280014790 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:52.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.264+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169973011 0 0) 0x7fc78c0b2a30 con 0x7fc78c13f710 2024-03-25T19:47:52.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.264+0000 7fc78affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:52.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.264+0000 7fc78affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:52.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.264+0000 7fc78affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:52.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.264+0000 7fc78affd640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:47:52.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.264+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc774003520 con 0x7fc78c13f710 2024-03-25T19:47:52.277 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.277+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579370360 0 0) 0x7fc78c0b28a0 con 0x7fc78c0a98f0 2024-03-25T19:47:52.277 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.277+0000 7fc78affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:52.277 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.277+0000 7fc78affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:52.277 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.277+0000 7fc78affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:52.277 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.277+0000 7fc78affd640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:47:52.277 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.277+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78c0b2a30 con 0x7fc78c0a98f0 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3068487047 0 0) 0x7fc774003520 con 0x7fc78c13f710 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 >> v1:172.21.15.169:6789/0 conn(0x7fc78c0a98f0 legacy=0x7fc78c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 >> v1:172.21.15.69:6789/0 conn(0x7fc78c1430c0 legacy=0x7fc78c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 1 monclient: found mon.noname-b 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0b44c0 con 0x7fc78c13f710 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 10 monclient: _finish_auth 0 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc780001690 con 0x7fc78c13f710 2024-03-25T19:47:52.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc78affd640 10 monclient: dump: 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc792713640 5 monclient: authenticate success, global_id 4529 2024-03-25T19:47:52.288 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.286+0000 7fc792713640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:52.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.293+0000 7fc78affd640 1 -- 172.21.15.69:0/280014790 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7800017f0 con 0x7fc78c13f710 2024-03-25T19:47:52.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.293+0000 7fc78affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:52.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.293+0000 7fc792713640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:52.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.293+0000 7fc792713640 4 set_mon_vals no callback set 2024-03-25T19:47:52.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.294+0000 7fc792713640 10 monclient: shutdown 2024-03-25T19:47:52.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.294+0000 7fc792713640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:52.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.294+0000 7fc792713640 1 -- 172.21.15.69:0/280014790 >> v1:172.21.15.69:6790/0 conn(0x7fc78c13f710 legacy=0x7fc78c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.294+0000 7fc792713640 1 -- 172.21.15.69:0/280014790 shutdown_connections 2024-03-25T19:47:52.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.294+0000 7fc792713640 1 -- 172.21.15.69:0/280014790 >> 172.21.15.69:0/280014790 conn(0x7fc78c0b7ea0 msgr2=0x7fc78c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:52.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.294+0000 7fc792713640 1 -- 172.21.15.69:0/280014790 shutdown_connections 2024-03-25T19:47:52.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.294+0000 7fc792713640 1 -- 172.21.15.69:0/280014790 wait complete. 2024-03-25T19:47:52.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 10 monclient: build_initial_monmap 2024-03-25T19:47:52.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 10 monclient: monmap: 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 1 librados: starting msgr at 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 1 librados: starting objecter 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 1 Processor -- start 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 1 -- start start 2024-03-25T19:47:52.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 1 librados: setting wanted keys 2024-03-25T19:47:52.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 1 librados: calling monclient init 2024-03-25T19:47:52.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.295+0000 7fc792713640 10 monclient: init 2024-03-25T19:47:52.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 10 monclient(hunting): picked mon.noname-c con 0x7fc78c0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 10 monclient(hunting): picked mon.noname-a con 0x7fc78c13f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 10 monclient(hunting): picked mon.noname-b con 0x7fc78c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c1d51f0 con 0x7fc78c13f710 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c1d7730 con 0x7fc78c1430c0 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c1d8360 con 0x7fc78c0a98f0 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:52.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.296+0000 7fc792713640 10 monclient(hunting): authenticate will time out at 5284.565165s 2024-03-25T19:47:52.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.298+0000 7fc790cb0640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc78c1430c0 0x7fc78c1e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:40824/0 (socket says 172.21.15.69:40824) 2024-03-25T19:47:52.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.298+0000 7fc790cb0640 1 -- 172.21.15.69:0/2230913525 learned_addr learned my addr 172.21.15.69:0/2230913525 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:52.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.300+0000 7fc788ff9640 10 client.?.objecter ms_handle_connect 0x7fc78c1430c0 2024-03-25T19:47:52.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.300+0000 7fc788ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:52.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.309+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249437815 0 0) 0x7fc78c1d7730 con 0x7fc78c1430c0 2024-03-25T19:47:52.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.309+0000 7fc788ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:52.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.309+0000 7fc788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:52.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.309+0000 7fc788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:52.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.309+0000 7fc788ff9640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:47:52.309 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.309+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778004eb0 con 0x7fc78c1430c0 2024-03-25T19:47:52.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.311+0000 7fc788ff9640 10 client.?.objecter ms_handle_connect 0x7fc78c0a98f0 2024-03-25T19:47:52.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.311+0000 7fc788ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:52.319 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.319+0000 7fc788ff9640 10 client.?.objecter ms_handle_connect 0x7fc78c13f710 2024-03-25T19:47:52.319 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.319+0000 7fc788ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:52.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.322+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3691525724 0 0) 0x7fc778004eb0 con 0x7fc78c1430c0 2024-03-25T19:47:52.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.322+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc78c1d7730 con 0x7fc78c1430c0 2024-03-25T19:47:52.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.322+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc784001c30 con 0x7fc78c1430c0 2024-03-25T19:47:52.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.322+0000 7fc788ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:52.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.322+0000 7fc788ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.322+0000 7fc788ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:52.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.322+0000 7fc788ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:52.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.323+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111305532 0 0) 0x7fc78c1d8360 con 0x7fc78c0a98f0 2024-03-25T19:47:52.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.323+0000 7fc788ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:52.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.323+0000 7fc788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:52.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.323+0000 7fc788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:52.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.323+0000 7fc788ff9640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:47:52.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.323+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778004eb0 con 0x7fc78c0a98f0 2024-03-25T19:47:52.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.332+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945445173 0 0) 0x7fc78c1d51f0 con 0x7fc78c13f710 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.332+0000 7fc788ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.332+0000 7fc788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78c1d8360 con 0x7fc78c13f710 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1145024200 0 0) 0x7fc78c1d7730 con 0x7fc78c1430c0 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 >> v1:172.21.15.69:6790/0 conn(0x7fc78c0a98f0 legacy=0x7fc78c1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 >> v1:172.21.15.69:6789/0 conn(0x7fc78c13f710 legacy=0x7fc78c138990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 1 monclient: found mon.b 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:52.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c1d84f0 con 0x7fc78c1430c0 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc788ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 5 monclient: authenticate success, global_id 4519 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 10 monclient: _renew_subs 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c1390a0 con 0x7fc78c1430c0 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 10 monclient: _renew_subs 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c138400 con 0x7fc78c1430c0 2024-03-25T19:47:52.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.333+0000 7fc792713640 1 librados: init done 2024-03-25T19:47:52.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.334+0000 7fc792713640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:52.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.334+0000 7fc792713640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:52.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.334+0000 7fc792713640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:52.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.334+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78c0ac040 con 0x7fc78c1430c0 2024-03-25T19:47:52.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.641+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc784003970 con 0x7fc78c1430c0 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.641+0000 7fc788ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.641+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc784003d40 con 0x7fc78c1430c0 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.641+0000 7fc788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.641+0000 7fc788ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.641+0000 7fc788ff9640 10 monclient: dump: 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:52.642 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:52.643 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:52.643 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:52.643 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.641+0000 7fc78a7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.642+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fc784003f60 con 0x7fc78c1430c0 2024-03-25T19:47:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.643+0000 7fc78bfff640 10 monclient: get_auth_request con 0x7fc77806de60 auth_method 0 2024-03-25T19:47:52.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.643+0000 7fc788ff9640 10 client.4519.objecter ms_handle_connect 0x7fc77806de60 2024-03-25T19:47:52.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.651+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fc784054a40 con 0x7fc78c1430c0 2024-03-25T19:47:52.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.651+0000 7fc788ff9640 10 client.4519.objecter ms_dispatch 0x7fc78c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:52.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.651+0000 7fc788ff9640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:52.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.651+0000 7fc788ff9640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:52.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.651+0000 7fc788ff9640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-25T19:47:52.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.651+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc784050550 con 0x7fc78c1430c0 2024-03-25T19:47:52.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.651+0000 7fc788ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:52.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.651+0000 7fc788ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:52.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.874+0000 7fc792713640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.874+0000 7fc792713640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.874+0000 7fc792713640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:47:52.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.874+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc78c1d78c0 con 0x7fc78c1430c0 2024-03-25T19:47:52.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.897+0000 7fc788ff9640 1 -- 172.21.15.69:0/2230913525 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fc784052480 con 0x7fc78c1430c0 2024-03-25T19:47:52.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.897+0000 7fc788ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.897+0000 7fc788ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:52.898 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:47:52.898 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 10 librados: watch_flush enter 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 10 librados: watch_flush exit 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 10 client.4519.objecter successfully canceled tick 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 >> v1:172.21.15.169:6800/3072548259 conn(0x7fc77806de60 legacy=0x7fc778070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 10 monclient: shutdown 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 >> v1:172.21.15.169:6789/0 conn(0x7fc78c1430c0 legacy=0x7fc78c1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 shutdown_connections 2024-03-25T19:47:52.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 >> 172.21.15.69:0/2230913525 conn(0x7fc78c0b7ea0 msgr2=0x7fc78c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:52.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.900+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 shutdown_connections 2024-03-25T19:47:52.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.901+0000 7fc792713640 1 -- 172.21.15.69:0/2230913525 wait complete. 2024-03-25T19:47:52.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:52.901+0000 7fc792713640 1 librados: shutdown 2024-03-25T19:47:52.912 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:59.719 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.830+0000 7facd9a57640 10 monclient: get_monmap_and_config 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.830+0000 7facd9a57640 10 monclient: build_initial_monmap 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.830+0000 7facd9a57640 10 monclient: monmap: 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:47:59.832115+0000 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:47:59.832115+0000 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:59.831 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:47:59.832 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:47:59.832 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.831+0000 7facd9a57640 1 Processor -- start 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 1 -- start start 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 10 monclient: init 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 10 monclient(hunting): picked mon.noname-a con 0x7facd40a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 10 monclient(hunting): picked mon.noname-b con 0x7facd413f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 10 monclient(hunting): picked mon.noname-c con 0x7facd41430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd4142520 con 0x7facd40a98f0 2024-03-25T19:47:59.833 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.832+0000 7facd9a57640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40b28a0 con 0x7facd41430c0 2024-03-25T19:47:59.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.833+0000 7facd9a57640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40b2a30 con 0x7facd413f710 2024-03-25T19:47:59.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.833+0000 7facd9a57640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:59.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.833+0000 7facd9a57640 10 monclient(hunting): authenticate will time out at 5292.101626s 2024-03-25T19:47:59.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.834+0000 7facd2ffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7facd40a98f0 0x7facd40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35406/0 (socket says 172.21.15.69:35406) 2024-03-25T19:47:59.834 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.834+0000 7facd2ffd640 1 -- 172.21.15.69:0/1611098098 learned_addr learned my addr 172.21.15.69:0/1611098098 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:59.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.845+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332636111 0 0) 0x7facd4142520 con 0x7facd40a98f0 2024-03-25T19:47:59.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.845+0000 7facd27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.845+0000 7facd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.845+0000 7facd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.845+0000 7facd27fc640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:47:59.845 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.845+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb4003520 con 0x7facd40a98f0 2024-03-25T19:47:59.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.846+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243848693 0 0) 0x7facd40b2a30 con 0x7facd413f710 2024-03-25T19:47:59.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.846+0000 7facd27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.846+0000 7facd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.846+0000 7facd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.846+0000 7facd27fc640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:47:59.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.846+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd4142520 con 0x7facd413f710 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 641256854 0 0) 0x7facd4142520 con 0x7facd413f710 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 >> v1:172.21.15.169:6789/0 conn(0x7facd41430c0 legacy=0x7facd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 >> v1:172.21.15.69:6789/0 conn(0x7facd40a98f0 legacy=0x7facd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 1 monclient: found mon.noname-b 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40b44c0 con 0x7facd413f710 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:59.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:59.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:59.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd9a57640 5 monclient: authenticate success, global_id 4535 2024-03-25T19:47:59.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.855+0000 7facd9a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:59.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.856+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7facbc001630 con 0x7facd413f710 2024-03-25T19:47:59.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.856+0000 7facd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.856+0000 7facd27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.856+0000 7facd27fc640 10 monclient: dump: 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:59.857 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.856+0000 7facd9a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:59.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.874+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facbc0018c0 con 0x7facd413f710 2024-03-25T19:47:59.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.874+0000 7facd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:59.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.874+0000 7facd27fc640 1 -- 172.21.15.69:0/1611098098 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7facbc001bd0 con 0x7facd413f710 2024-03-25T19:47:59.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.874+0000 7facd9a57640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:59.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.874+0000 7facd9a57640 4 set_mon_vals no callback set 2024-03-25T19:47:59.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.874+0000 7facd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.874+0000 7facd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.874+0000 7facd27fc640 10 monclient: dump: 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:59.875 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:59.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 10 monclient: shutdown 2024-03-25T19:47:59.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:59.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 1 -- 172.21.15.69:0/1611098098 >> v1:172.21.15.69:6790/0 conn(0x7facd413f710 legacy=0x7facd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 1 -- 172.21.15.69:0/1611098098 shutdown_connections 2024-03-25T19:47:59.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 1 -- 172.21.15.69:0/1611098098 >> 172.21.15.69:0/1611098098 conn(0x7facd40b7ea0 msgr2=0x7facd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:59.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 1 -- 172.21.15.69:0/1611098098 shutdown_connections 2024-03-25T19:47:59.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 1 -- 172.21.15.69:0/1611098098 wait complete. 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 10 monclient: build_initial_monmap 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.875+0000 7facd9a57640 10 monclient: monmap: 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.876+0000 7facd9a57640 1 librados: starting msgr at 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.876+0000 7facd9a57640 1 librados: starting objecter 2024-03-25T19:47:59.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.876+0000 7facd9a57640 1 Processor -- start 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.876+0000 7facd9a57640 1 -- start start 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.876+0000 7facd9a57640 1 librados: setting wanted keys 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.876+0000 7facd9a57640 1 librados: calling monclient init 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.876+0000 7facd9a57640 10 monclient: init 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 10 monclient(hunting): picked mon.noname-a con 0x7facd40a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 10 monclient(hunting): picked mon.noname-b con 0x7facd413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 10 monclient(hunting): picked mon.noname-c con 0x7facd41430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40b1730 con 0x7facd40a98f0 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40b0920 con 0x7facd413f710 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd41dae10 con 0x7facd41430c0 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:59.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.877+0000 7facd9a57640 10 monclient(hunting): authenticate will time out at 5292.145683s 2024-03-25T19:47:59.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.882+0000 7facd2ffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7facd40a98f0 0x7facd41d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35464/0 (socket says 172.21.15.69:35464) 2024-03-25T19:47:59.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.882+0000 7facd2ffd640 1 -- 172.21.15.69:0/1966847145 learned_addr learned my addr 172.21.15.69:0/1966847145 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:47:59.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.883+0000 7facd0ff9640 10 client.?.objecter ms_handle_connect 0x7facd40a98f0 2024-03-25T19:47:59.883 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.883+0000 7facd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.891+0000 7facd0ff9640 10 client.?.objecter ms_handle_connect 0x7facd413f710 2024-03-25T19:47:59.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.891+0000 7facd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.892+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309110163 0 0) 0x7facd40b1730 con 0x7facd40a98f0 2024-03-25T19:47:59.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.892+0000 7facd0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.892+0000 7facd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.892+0000 7facd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.892+0000 7facd0ff9640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:47:59.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.892+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8004eb0 con 0x7facd40a98f0 2024-03-25T19:47:59.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.902+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1293362358 0 0) 0x7facb8004eb0 con 0x7facd40a98f0 2024-03-25T19:47:59.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.902+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facd40b1730 con 0x7facd40a98f0 2024-03-25T19:47:59.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.902+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7facc4001950 con 0x7facd40a98f0 2024-03-25T19:47:59.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.902+0000 7facd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.902+0000 7facd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.902+0000 7facd0ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:47:59.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.902+0000 7facd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:59.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.903+0000 7facd0ff9640 10 client.?.objecter ms_handle_connect 0x7facd41430c0 2024-03-25T19:47:59.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.903+0000 7facd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.911+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142420100 0 0) 0x7facd40b0920 con 0x7facd413f710 2024-03-25T19:47:59.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.911+0000 7facd0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.911 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.911+0000 7facd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.911+0000 7facd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.911+0000 7facd0ff9640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:47:59.912 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.911+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8004eb0 con 0x7facd413f710 2024-03-25T19:47:59.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 851431982 0 0) 0x7facd40b1730 con 0x7facd40a98f0 2024-03-25T19:47:59.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 >> v1:172.21.15.69:6790/0 conn(0x7facd41430c0 legacy=0x7facd40b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 >> v1:172.21.15.169:6789/0 conn(0x7facd413f710 legacy=0x7facd40acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:59.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 1 monclient: found mon.a 2024-03-25T19:47:59.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd41d81b0 con 0x7facd40a98f0 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 5 monclient: authenticate success, global_id 4557 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 10 monclient: _renew_subs 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd41da8f0 con 0x7facd40a98f0 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 10 monclient: _renew_subs 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facd414dfa0 con 0x7facd40a98f0 2024-03-25T19:47:59.914 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.913+0000 7facd9a57640 1 librados: init done 2024-03-25T19:47:59.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.914+0000 7facd9a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:59.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.914+0000 7facd9a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:59.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.914+0000 7facd9a57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:47:59.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:47:59.914+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd41dafa0 con 0x7facd40a98f0 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.216+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc40047e0 con 0x7facd40a98f0 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.216+0000 7facd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.216+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7facc4004b70 con 0x7facd40a98f0 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.216+0000 7facd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.216+0000 7facd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.216+0000 7facd0ff9640 10 monclient: dump: 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:00.217 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:00.218 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:00.218 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:00.218 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:00.218 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:00.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.216+0000 7facd1ffb640 4 set_mon_vals no callback set 2024-03-25T19:48:00.219 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.217+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7facc4004d90 con 0x7facd40a98f0 2024-03-25T19:48:00.219 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.218+0000 7faccbfff640 10 monclient: get_auth_request con 0x7facb806dd40 auth_method 0 2024-03-25T19:48:00.219 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.218+0000 7facd0ff9640 10 client.4557.objecter ms_handle_connect 0x7facb806dd40 2024-03-25T19:48:00.224 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.224+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7facc4054840 con 0x7facd40a98f0 2024-03-25T19:48:00.225 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.224+0000 7facd0ff9640 10 client.4557.objecter ms_dispatch 0x7facd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:00.225 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.224+0000 7facd0ff9640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:00.225 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.224+0000 7facd0ff9640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:00.225 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.224+0000 7facd0ff9640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-25T19:48:00.225 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.224+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facc4050350 con 0x7facd40a98f0 2024-03-25T19:48:00.225 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.224+0000 7facd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:00.225 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.224+0000 7facd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:00.472 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.471+0000 7facd9a57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.472 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.472+0000 7facd9a57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.472 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.472+0000 7facd9a57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:48:00.472 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.472+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facd41d8340 con 0x7facd40a98f0 2024-03-25T19:48:00.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.497+0000 7facd0ff9640 1 -- 172.21.15.69:0/1966847145 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7facd41d8340 con 0x7facd40a98f0 2024-03-25T19:48:00.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.497+0000 7facd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.497+0000 7facd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:00.498 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:48:00.498 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:00.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.499+0000 7facd9a57640 10 librados: watch_flush enter 2024-03-25T19:48:00.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 10 librados: watch_flush exit 2024-03-25T19:48:00.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-25T19:48:00.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 10 client.4557.objecter successfully canceled tick 2024-03-25T19:48:00.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 >> v1:172.21.15.169:6800/3072548259 conn(0x7facb806dd40 legacy=0x7facb8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:00.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 10 monclient: shutdown 2024-03-25T19:48:00.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:00.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 >> v1:172.21.15.69:6789/0 conn(0x7facd40a98f0 legacy=0x7facd41d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:00.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 shutdown_connections 2024-03-25T19:48:00.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 >> 172.21.15.69:0/1966847145 conn(0x7facd40b7ea0 msgr2=0x7facd4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:00.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 shutdown_connections 2024-03-25T19:48:00.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 1 -- 172.21.15.69:0/1966847145 wait complete. 2024-03-25T19:48:00.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:00.500+0000 7facd9a57640 1 librados: shutdown 2024-03-25T19:48:00.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:07.320 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:07.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.434+0000 7fb0114c8640 10 monclient: get_monmap_and_config 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.434+0000 7fb0114c8640 10 monclient: build_initial_monmap 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.434+0000 7fb0114c8640 10 monclient: monmap: 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:48:07.435797+0000 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:48:07.435797+0000 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:48:07.435 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.435+0000 7fb0114c8640 1 Processor -- start 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.435+0000 7fb0114c8640 1 -- start start 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.435+0000 7fb0114c8640 10 monclient: init 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 10 monclient(hunting): picked mon.noname-b con 0x7fb00c0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 10 monclient(hunting): picked mon.noname-c con 0x7fb00c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 10 monclient(hunting): picked mon.noname-a con 0x7fb00c1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c142520 con 0x7fb00c1430c0 2024-03-25T19:48:07.438 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c0b28a0 con 0x7fb00c13f710 2024-03-25T19:48:07.438 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c0b2a30 con 0x7fb00c0a98f0 2024-03-25T19:48:07.438 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:07.438 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.436+0000 7fb0114c8640 10 monclient(hunting): authenticate will time out at 5299.705088s 2024-03-25T19:48:07.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.443+0000 7fb003fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb00c13f710 0x7fb00c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:46082/0 (socket says 172.21.15.69:46082) 2024-03-25T19:48:07.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.443+0000 7fb003fff640 1 -- 172.21.15.69:0/1554905541 learned_addr learned my addr 172.21.15.69:0/1554905541 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:07.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.464+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526142597 0 0) 0x7fb00c0b28a0 con 0x7fb00c13f710 2024-03-25T19:48:07.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.464+0000 7fb00a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.464+0000 7fb00a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.464+0000 7fb00a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.464+0000 7fb00a7fc640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:48:07.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.464+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff8003520 con 0x7fb00c13f710 2024-03-25T19:48:07.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.470+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726758742 0 0) 0x7fb00c0b2a30 con 0x7fb00c0a98f0 2024-03-25T19:48:07.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.470+0000 7fb00a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.470+0000 7fb00a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.470+0000 7fb00a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.470+0000 7fb00a7fc640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:48:07.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.470+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb00c0b28a0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1248523432 0 0) 0x7fb00c0b28a0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 >> v1:172.21.15.169:6789/0 conn(0x7fb00c13f710 legacy=0x7fb00c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 >> v1:172.21.15.69:6789/0 conn(0x7fb00c1430c0 legacy=0x7fb00c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c0b44c0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:07.486 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7faffc001670 con 0x7fb00c0a98f0 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb00a7fc640 10 monclient: dump: 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb0114c8640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:48:07.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.485+0000 7fb0114c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:07.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.496+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faffc0017d0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.496+0000 7fb00a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:07.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.496+0000 7fb0114c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:07.496 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.496+0000 7fb0114c8640 4 set_mon_vals no callback set 2024-03-25T19:48:07.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb00a7fc640 1 -- 172.21.15.69:0/1554905541 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7faffc001c20 con 0x7fb00c0a98f0 2024-03-25T19:48:07.498 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb00a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb00a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb00a7fc640 10 monclient: dump: 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb0114c8640 10 monclient: shutdown 2024-03-25T19:48:07.499 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb0114c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb0114c8640 1 -- 172.21.15.69:0/1554905541 >> v1:172.21.15.69:6790/0 conn(0x7fb00c0a98f0 legacy=0x7fb00c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb0114c8640 1 -- 172.21.15.69:0/1554905541 shutdown_connections 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb0114c8640 1 -- 172.21.15.69:0/1554905541 >> 172.21.15.69:0/1554905541 conn(0x7fb00c0b7ea0 msgr2=0x7fb00c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb0114c8640 1 -- 172.21.15.69:0/1554905541 shutdown_connections 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.497+0000 7fb0114c8640 1 -- 172.21.15.69:0/1554905541 wait complete. 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 10 monclient: build_initial_monmap 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 10 monclient: monmap: 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:48:07.500 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 1 librados: starting msgr at 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 1 librados: starting objecter 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 1 Processor -- start 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 1 -- start start 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 1 librados: setting wanted keys 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 1 librados: calling monclient init 2024-03-25T19:48:07.501 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.498+0000 7fb0114c8640 10 monclient: init 2024-03-25T19:48:07.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:07.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:07.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 10 monclient(hunting): picked mon.noname-c con 0x7fb00c0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:07.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 10 monclient(hunting): picked mon.noname-a con 0x7fb00c13f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:07.502 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 10 monclient(hunting): picked mon.noname-b con 0x7fb00c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:07.503 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c12db70 con 0x7fb00c13f710 2024-03-25T19:48:07.503 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c13b8d0 con 0x7fb00c1430c0 2024-03-25T19:48:07.503 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c139d20 con 0x7fb00c0a98f0 2024-03-25T19:48:07.503 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:07.503 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.499+0000 7fb0114c8640 10 monclient(hunting): authenticate will time out at 5299.768111s 2024-03-25T19:48:07.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.504+0000 7fb00affd640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fb00c0a98f0 0x7fb00c13bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:42816/0 (socket says 172.21.15.69:42816) 2024-03-25T19:48:07.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.504+0000 7fb00affd640 1 -- 172.21.15.69:0/1528306005 learned_addr learned my addr 172.21.15.69:0/1528306005 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:07.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.504+0000 7fb0037fe640 10 client.?.objecter ms_handle_connect 0x7fb00c0a98f0 2024-03-25T19:48:07.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.504+0000 7fb0037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.516+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607200993 0 0) 0x7fb00c139d20 con 0x7fb00c0a98f0 2024-03-25T19:48:07.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.516+0000 7fb0037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.516+0000 7fb0037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.516+0000 7fb0037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.516+0000 7fb0037fe640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:48:07.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.516+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafec004eb0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.516+0000 7fb0037fe640 10 client.?.objecter ms_handle_connect 0x7fb00c13f710 2024-03-25T19:48:07.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.516+0000 7fb0037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.520+0000 7fb0037fe640 10 client.?.objecter ms_handle_connect 0x7fb00c1430c0 2024-03-25T19:48:07.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.520+0000 7fb0037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.524+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266567584 0 0) 0x7fb00c12db70 con 0x7fb00c13f710 2024-03-25T19:48:07.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.524+0000 7fb0037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.525 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.524+0000 7fb0037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.525 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.524+0000 7fb0037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.525 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.524+0000 7fb0037fe640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:48:07.525 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.524+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb00c139d20 con 0x7fb00c13f710 2024-03-25T19:48:07.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.527+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3404431155 0 0) 0x7fafec004eb0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.527+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb00c12db70 con 0x7fb00c0a98f0 2024-03-25T19:48:07.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.527+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7faffc001c20 con 0x7fb00c0a98f0 2024-03-25T19:48:07.527 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.527+0000 7fb0037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.527+0000 7fb0037fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.527+0000 7fb0037fe640 10 monclient(hunting): dump: 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:07.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.527+0000 7fb0037fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:07.540 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 50267092 0 0) 0x7fb00c12db70 con 0x7fb00c0a98f0 2024-03-25T19:48:07.540 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 >> v1:172.21.15.169:6789/0 conn(0x7fb00c1430c0 legacy=0x7fb00c12d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.540 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 >> v1:172.21.15.69:6789/0 conn(0x7fb00c13f710 legacy=0x7fb00c12cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.540 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:07.540 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 1 monclient: found mon.c 2024-03-25T19:48:07.540 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:07.540 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:07.540 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c13c400 con 0x7fb00c0a98f0 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 5 monclient: authenticate success, global_id 4544 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 10 monclient: _renew_subs 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c13c5f0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 10 monclient: _renew_subs 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c13a700 con 0x7fb00c0a98f0 2024-03-25T19:48:07.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.539+0000 7fb0114c8640 1 librados: init done 2024-03-25T19:48:07.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.540+0000 7fb0114c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.540+0000 7fb0114c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.540+0000 7fb0114c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:07.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.540+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c139eb0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.542+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faffc0038a0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.542+0000 7fb0037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:07.542 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.542+0000 7fb009ffb640 4 set_mon_vals no callback set 2024-03-25T19:48:07.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.561+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7faffc003bf0 con 0x7fb00c0a98f0 2024-03-25T19:48:07.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.561+0000 7fb0037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.561 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.561+0000 7fb0037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.561+0000 7fb0037fe640 10 monclient: dump: 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:07.562 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:07.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.562+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7faffc003d50 con 0x7fb00c0a98f0 2024-03-25T19:48:07.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.563+0000 7fb003fff640 10 monclient: get_auth_request con 0x7fafec06dd50 auth_method 0 2024-03-25T19:48:07.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.563+0000 7fb0037fe640 10 client.4544.objecter ms_handle_connect 0x7fafec06dd50 2024-03-25T19:48:07.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.565+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7faffc054970 con 0x7fb00c0a98f0 2024-03-25T19:48:07.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.565+0000 7fb0037fe640 10 client.4544.objecter ms_dispatch 0x7fb00c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:07.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.565+0000 7fb0037fe640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:07.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.565+0000 7fb0037fe640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:07.566 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.565+0000 7fb0037fe640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-25T19:48:07.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.839+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faffc050480 con 0x7fb00c0a98f0 2024-03-25T19:48:07.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.839+0000 7fb0037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.840 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:07.839+0000 7fb0037fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:08.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.072+0000 7fb0114c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:08.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.072+0000 7fb0114c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:08.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.072+0000 7fb0114c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:08.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.072+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb00c13b6f0 con 0x7fb00c0a98f0 2024-03-25T19:48:08.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.093+0000 7fb0037fe640 1 -- 172.21.15.69:0/1528306005 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7faffc0523b0 con 0x7fb00c0a98f0 2024-03-25T19:48:08.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.093+0000 7fb0037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:08.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.093+0000 7fb0037fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:08.094 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:48:08.094 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:08.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 10 librados: watch_flush enter 2024-03-25T19:48:08.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 10 librados: watch_flush exit 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 10 client.4544.objecter successfully canceled tick 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 >> v1:172.21.15.169:6800/3072548259 conn(0x7fafec06dd50 legacy=0x7fafec070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 10 monclient: shutdown 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 >> v1:172.21.15.69:6790/0 conn(0x7fb00c0a98f0 legacy=0x7fb00c13bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 shutdown_connections 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 >> 172.21.15.69:0/1528306005 conn(0x7fb00c0b7ea0 msgr2=0x7fb00c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 shutdown_connections 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 1 -- 172.21.15.69:0/1528306005 wait complete. 2024-03-25T19:48:08.097 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:08.096+0000 7fb0114c8640 1 librados: shutdown 2024-03-25T19:48:08.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:08.174 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:46.138 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:08.229 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:46.193 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:08.270 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:46.234 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:08.285 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:46.249 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:08.288 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:46.242 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:08.295 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:46.259 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:08.296 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:46.250 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:08.319 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:46.273 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:08.321 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:46.285 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:14.914 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:15.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.031+0000 7f96a8682640 10 monclient: get_monmap_and_config 2024-03-25T19:48:15.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.031+0000 7f96a8682640 10 monclient: build_initial_monmap 2024-03-25T19:48:15.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.031+0000 7f96a8682640 10 monclient: monmap: 2024-03-25T19:48:15.031 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:15.031 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:15.031 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:48:15.032615+0000 2024-03-25T19:48:15.032 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:48:15.032615+0000 2024-03-25T19:48:15.032 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:15.032 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:15.032 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:15.032 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:48:15.032 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:48:15.032 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 1 Processor -- start 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 1 -- start start 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 10 monclient: init 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 10 monclient(hunting): picked mon.noname-a con 0x7f96a00b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 10 monclient(hunting): picked mon.noname-c con 0x7f96a00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 10 monclient(hunting): picked mon.noname-b con 0x7f96a014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:15.033 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a00af940 con 0x7f96a00b4d60 2024-03-25T19:48:15.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a00b0a70 con 0x7f96a00b1730 2024-03-25T19:48:15.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a00b0d50 con 0x7f96a014be90 2024-03-25T19:48:15.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:15.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.032+0000 7f96a8682640 10 monclient(hunting): authenticate will time out at 5307.301531s 2024-03-25T19:48:15.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.036+0000 7f96a641e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f96a00b4d60 0x7f96a013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33550/0 (socket says 172.21.15.69:33550) 2024-03-25T19:48:15.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.036+0000 7f96a641e640 1 -- 172.21.15.69:0/3642349067 learned_addr learned my addr 172.21.15.69:0/3642349067 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:15.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.051+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202326197 0 0) 0x7f96a00b0d50 con 0x7f96a014be90 2024-03-25T19:48:15.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.051+0000 7f96a541c640 10 monclient(hunting): _negotiate 2024-03-25T19:48:15.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.051+0000 7f96a541c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:15.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.051+0000 7f96a541c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:15.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.051+0000 7f96a541c640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:48:15.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.051+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f968c003520 con 0x7f96a014be90 2024-03-25T19:48:15.053 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.053+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609458794 0 0) 0x7f96a00af940 con 0x7f96a00b4d60 2024-03-25T19:48:15.053 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.053+0000 7f96a541c640 10 monclient(hunting): _negotiate 2024-03-25T19:48:15.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.053+0000 7f96a541c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:15.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.053+0000 7f96a541c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:15.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.053+0000 7f96a541c640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:48:15.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.053+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a00b0d50 con 0x7f96a00b4d60 2024-03-25T19:48:15.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.058+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144705753 0 0) 0x7f96a00b0a70 con 0x7f96a00b1730 2024-03-25T19:48:15.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.058+0000 7f96a541c640 10 monclient(hunting): _negotiate 2024-03-25T19:48:15.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.058+0000 7f96a541c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:15.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.058+0000 7f96a541c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:15.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.058+0000 7f96a541c640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:48:15.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.058+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a00af940 con 0x7f96a00b1730 2024-03-25T19:48:15.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1092939716 0 0) 0x7f96a00b0d50 con 0x7f96a00b4d60 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 >> v1:172.21.15.69:6790/0 conn(0x7f96a014be90 legacy=0x7f96a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 >> v1:172.21.15.169:6789/0 conn(0x7f96a00b1730 legacy=0x7f96a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 10 monclient: _finish_hunting 0 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 1 monclient: found mon.noname-a 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a00b1030 con 0x7f96a00b4d60 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 10 monclient: _finish_auth 0 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 10 monclient: _check_auth_tickets 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a541c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a8682640 5 monclient: authenticate success, global_id 4566 2024-03-25T19:48:15.065 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.064+0000 7f96a8682640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:15.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.065+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f969c001630 con 0x7f96a00b4d60 2024-03-25T19:48:15.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.065+0000 7f96a541c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:15.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.065+0000 7f96a541c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:15.066 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.065+0000 7f96a541c640 10 monclient: dump: 2024-03-25T19:48:15.066 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:15.066 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:15.066 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:15.067 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:15.067 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:15.067 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:15.067 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:15.067 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:15.067 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:15.067 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:15.067 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.065+0000 7f96a8682640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.086+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f969c001880 con 0x7f96a00b4d60 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.086+0000 7f96a541c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.086+0000 7f96a541c640 1 -- 172.21.15.69:0/3642349067 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f969c001c10 con 0x7f96a00b4d60 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.086+0000 7f96a8682640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.086+0000 7f96a8682640 4 set_mon_vals no callback set 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.086+0000 7f96a541c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.086+0000 7f96a541c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.086+0000 7f96a541c640 10 monclient: dump: 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:15.087 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:15.088 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:15.088 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:15.088 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:15.088 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:15.088 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:15.088 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:15.088 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:15.088 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:15.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.087+0000 7f96a8682640 10 monclient: shutdown 2024-03-25T19:48:15.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.087+0000 7f96a8682640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.087+0000 7f96a8682640 1 -- 172.21.15.69:0/3642349067 >> v1:172.21.15.69:6789/0 conn(0x7f96a00b4d60 legacy=0x7f96a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.088+0000 7f96a8682640 1 -- 172.21.15.69:0/3642349067 shutdown_connections 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.088+0000 7f96a8682640 1 -- 172.21.15.69:0/3642349067 >> 172.21.15.69:0/3642349067 conn(0x7f96a00617a0 msgr2=0x7f96a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.088+0000 7f96a8682640 1 -- 172.21.15.69:0/3642349067 shutdown_connections 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.088+0000 7f96a8682640 1 -- 172.21.15.69:0/3642349067 wait complete. 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.088+0000 7f96a8682640 10 monclient: build_initial_monmap 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.088+0000 7f96a8682640 10 monclient: monmap: 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:48:15.091 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 librados: starting msgr at 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 librados: starting objecter 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 Processor -- start 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 -- start start 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 librados: setting wanted keys 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 librados: calling monclient init 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 10 monclient: init 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 10 monclient(hunting): picked mon.noname-b con 0x7f96a00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 10 monclient(hunting): picked mon.noname-a con 0x7f96a00b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 10 monclient(hunting): picked mon.noname-c con 0x7f96a014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:15.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a01d77a0 con 0x7f96a00b4d60 2024-03-25T19:48:15.093 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a01d8670 con 0x7f96a00b1730 2024-03-25T19:48:15.093 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a00aad20 con 0x7f96a014be90 2024-03-25T19:48:15.093 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:15.093 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.089+0000 7f96a8682640 10 monclient(hunting): authenticate will time out at 5307.358540s 2024-03-25T19:48:15.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.096+0000 7f96a6c1f640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f96a014be90 0x7f96a0137f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:40610/0 (socket says 172.21.15.69:40610) 2024-03-25T19:48:15.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.096+0000 7f96a6c1f640 1 -- 172.21.15.69:0/2697115866 learned_addr learned my addr 172.21.15.69:0/2697115866 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:15.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.096+0000 7f9696ffd640 10 client.?.objecter ms_handle_connect 0x7f96a014be90 2024-03-25T19:48:15.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.096+0000 7f9696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:15.102 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.102+0000 7f9696ffd640 10 client.?.objecter ms_handle_connect 0x7f96a00b1730 2024-03-25T19:48:15.102 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.102+0000 7f9696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:15.105 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.105+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199695295 0 0) 0x7f96a00aad20 con 0x7f96a014be90 2024-03-25T19:48:15.105 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.105+0000 7f9696ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:15.105 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.105+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:15.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.105+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:15.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.105+0000 7f9696ffd640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:48:15.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.105+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9684004eb0 con 0x7f96a014be90 2024-03-25T19:48:15.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.111+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296635182 0 0) 0x7f96a01d8670 con 0x7f96a00b1730 2024-03-25T19:48:15.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.111+0000 7f9696ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:15.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.111+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:15.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.111+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:15.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.111+0000 7f9696ffd640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:48:15.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.111+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a00aad20 con 0x7f96a00b1730 2024-03-25T19:48:15.112 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.112+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1059419973 0 0) 0x7f9684004eb0 con 0x7f96a014be90 2024-03-25T19:48:15.112 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.112+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96a01d8670 con 0x7f96a014be90 2024-03-25T19:48:15.112 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.112+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9698001c30 con 0x7f96a014be90 2024-03-25T19:48:15.112 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.112+0000 7f9696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.112+0000 7f9696ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.112+0000 7f9696ffd640 10 monclient(hunting): dump: 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:15.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.112+0000 7f9696ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:15.114 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.113+0000 7f9696ffd640 10 client.?.objecter ms_handle_connect 0x7f96a00b4d60 2024-03-25T19:48:15.114 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.113+0000 7f9696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:15.118 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1211448722 0 0) 0x7f96a01d8670 con 0x7f96a014be90 2024-03-25T19:48:15.118 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 >> v1:172.21.15.169:6789/0 conn(0x7f96a00b1730 legacy=0x7f96a013b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.118 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 >> v1:172.21.15.69:6789/0 conn(0x7f96a00b4d60 legacy=0x7f96a01d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.118 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:15.118 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 1 monclient: found mon.c 2024-03-25T19:48:15.118 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:15.118 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a01d7a40 con 0x7f96a014be90 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 10 monclient: _finish_auth 0 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f9696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f96a8682640 5 monclient: authenticate success, global_id 4550 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f96a8682640 10 monclient: _renew_subs 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f96a8682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.117+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a0138680 con 0x7f96a014be90 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 10 monclient: _renew_subs 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a00a98c0 con 0x7f96a014be90 2024-03-25T19:48:15.119 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 1 librados: init done 2024-03-25T19:48:15.120 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:15.120 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:15.120 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:15.120 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.118+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a0138980 con 0x7f96a014be90 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.411+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9698003970 con 0x7f96a014be90 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.411+0000 7f9696ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.411+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9698003d00 con 0x7f96a014be90 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.411+0000 7f9696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.411+0000 7f9696ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.411+0000 7f9696ffd640 10 monclient: dump: 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:15.412 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:15.413 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:15.413 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:15.413 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:15.413 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:15.413 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:15.413 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:15.413 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.411+0000 7f96a4c1b640 4 set_mon_vals no callback set 2024-03-25T19:48:15.414 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.412+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f9698003f20 con 0x7f96a014be90 2024-03-25T19:48:15.414 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.413+0000 7f96a641e640 10 monclient: get_auth_request con 0x7f968406dda0 auth_method 0 2024-03-25T19:48:15.414 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.413+0000 7f9696ffd640 10 client.4550.objecter ms_handle_connect 0x7f968406dda0 2024-03-25T19:48:15.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.420+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f9698054a00 con 0x7f96a014be90 2024-03-25T19:48:15.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.420+0000 7f9696ffd640 10 client.4550.objecter ms_dispatch 0x7f96a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:15.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.420+0000 7f9696ffd640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:15.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.420+0000 7f9696ffd640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:15.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.420+0000 7f9696ffd640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T19:48:15.420 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.420+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9698050510 con 0x7f96a014be90 2024-03-25T19:48:15.421 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.420+0000 7f9696ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:15.421 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.420+0000 7f9696ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:15.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.663+0000 7f96a8682640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.663+0000 7f96a8682640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.663+0000 7f96a8682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:15.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.663+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96a01d7bd0 con 0x7f96a014be90 2024-03-25T19:48:15.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.691+0000 7f9696ffd640 1 -- 172.21.15.69:0/2697115866 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f9698052440 con 0x7f96a014be90 2024-03-25T19:48:15.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.691+0000 7f9696ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.691+0000 7f9696ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:15.691 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:48:15.692 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:15.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 10 librados: watch_flush enter 2024-03-25T19:48:15.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 10 librados: watch_flush exit 2024-03-25T19:48:15.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T19:48:15.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 10 client.4550.objecter successfully canceled tick 2024-03-25T19:48:15.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 >> v1:172.21.15.169:6800/3072548259 conn(0x7f968406dda0 legacy=0x7f9684070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 10 monclient: shutdown 2024-03-25T19:48:15.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:15.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 >> v1:172.21.15.69:6790/0 conn(0x7f96a014be90 legacy=0x7f96a0137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 shutdown_connections 2024-03-25T19:48:15.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 >> 172.21.15.69:0/2697115866 conn(0x7f96a00617a0 msgr2=0x7f96a00b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:15.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 shutdown_connections 2024-03-25T19:48:15.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 1 -- 172.21.15.69:0/2697115866 wait complete. 2024-03-25T19:48:15.695 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:15.693+0000 7f96a8682640 1 librados: shutdown 2024-03-25T19:48:15.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:22.512 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:22.624 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.624+0000 7f3ea960f640 10 monclient: get_monmap_and_config 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.624+0000 7f3ea960f640 10 monclient: build_initial_monmap 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.624+0000 7f3ea960f640 10 monclient: monmap: 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:48:22.625857+0000 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:48:22.625857+0000 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:48:22.625 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:22.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.625+0000 7f3ea960f640 1 Processor -- start 2024-03-25T19:48:22.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.625+0000 7f3ea960f640 1 -- start start 2024-03-25T19:48:22.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.625+0000 7f3ea960f640 10 monclient: init 2024-03-25T19:48:22.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.625+0000 7f3ea960f640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:22.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.625+0000 7f3ea960f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:22.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea960f640 10 monclient(hunting): picked mon.noname-b con 0x7f3ea40a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:22.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea960f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ea413f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:22.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea960f640 10 monclient(hunting): picked mon.noname-a con 0x7f3ea41430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:22.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea960f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea4142520 con 0x7f3ea41430c0 2024-03-25T19:48:22.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea960f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea40b28a0 con 0x7f3ea413f710 2024-03-25T19:48:22.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea960f640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea40b2a30 con 0x7f3ea40a98f0 2024-03-25T19:48:22.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea960f640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:22.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea960f640 10 monclient(hunting): authenticate will time out at 5314.894720s 2024-03-25T19:48:22.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea2ffd640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f3ea40a98f0 0x7f3ea40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:54096/0 (socket says 172.21.15.69:54096) 2024-03-25T19:48:22.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.626+0000 7f3ea2ffd640 1 -- 172.21.15.69:0/905307476 learned_addr learned my addr 172.21.15.69:0/905307476 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:22.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.704+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839110204 0 0) 0x7f3ea4142520 con 0x7f3ea41430c0 2024-03-25T19:48:22.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.704+0000 7f3ea1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.704+0000 7f3ea1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.704+0000 7f3ea1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.704+0000 7f3ea1ffb640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:48:22.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.704+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e8c003520 con 0x7f3ea41430c0 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397643092 0 0) 0x7f3e8c003520 con 0x7f3ea41430c0 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 >> v1:172.21.15.69:6790/0 conn(0x7f3ea40a98f0 legacy=0x7f3ea40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 >> v1:172.21.15.169:6789/0 conn(0x7f3ea413f710 legacy=0x7f3ea41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 1 monclient: found mon.noname-a 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea40b44c0 con 0x7f3ea41430c0 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:22.714 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3e9c001a10 con 0x7f3ea41430c0 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea960f640 5 monclient: authenticate success, global_id 4572 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea960f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea1ffb640 10 monclient: dump: 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:22.715 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:22.716 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:22.716 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:22.716 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:22.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.713+0000 7f3ea960f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:22.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e9c001b70 con 0x7f3ea41430c0 2024-03-25T19:48:22.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:22.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea1ffb640 1 -- 172.21.15.69:0/905307476 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3e9c001fc0 con 0x7f3ea41430c0 2024-03-25T19:48:22.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea960f640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:22.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea960f640 4 set_mon_vals no callback set 2024-03-25T19:48:22.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:22.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea1ffb640 10 monclient: dump: 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:22.726 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:22.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea960f640 10 monclient: shutdown 2024-03-25T19:48:22.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea960f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:22.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.725+0000 7f3ea960f640 1 -- 172.21.15.69:0/905307476 >> v1:172.21.15.69:6789/0 conn(0x7f3ea41430c0 legacy=0x7f3ea414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.726+0000 7f3ea960f640 1 -- 172.21.15.69:0/905307476 shutdown_connections 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.726+0000 7f3ea960f640 1 -- 172.21.15.69:0/905307476 >> 172.21.15.69:0/905307476 conn(0x7f3ea40b7ea0 msgr2=0x7f3ea40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.726+0000 7f3ea960f640 1 -- 172.21.15.69:0/905307476 shutdown_connections 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.726+0000 7f3ea960f640 1 -- 172.21.15.69:0/905307476 wait complete. 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.726+0000 7f3ea960f640 10 monclient: build_initial_monmap 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.726+0000 7f3ea960f640 10 monclient: monmap: 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:48:22.729 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 1 librados: starting msgr at 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 1 librados: starting objecter 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 1 Processor -- start 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 1 -- start start 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 1 librados: setting wanted keys 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 1 librados: calling monclient init 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 10 monclient: init 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.727+0000 7f3ea960f640 10 monclient(hunting): picked mon.noname-b con 0x7f3ea40a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.728+0000 7f3ea960f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ea413f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.728+0000 7f3ea960f640 10 monclient(hunting): picked mon.noname-a con 0x7f3ea41430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.728+0000 7f3ea960f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea4137f10 con 0x7f3ea41430c0 2024-03-25T19:48:22.730 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.728+0000 7f3ea960f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea41dc070 con 0x7f3ea40a98f0 2024-03-25T19:48:22.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.728+0000 7f3ea960f640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea4137100 con 0x7f3ea413f710 2024-03-25T19:48:22.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.728+0000 7f3ea960f640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:22.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.728+0000 7f3ea960f640 10 monclient(hunting): authenticate will time out at 5314.996730s 2024-03-25T19:48:22.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.736+0000 7f3ea37fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3ea41430c0 0x7f3ea41db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55814/0 (socket says 172.21.15.69:55814) 2024-03-25T19:48:22.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.736+0000 7f3ea37fe640 1 -- 172.21.15.69:0/1239831788 learned_addr learned my addr 172.21.15.69:0/1239831788 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:22.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.743+0000 7f3e8b7fe640 10 client.?.objecter ms_handle_connect 0x7f3ea41430c0 2024-03-25T19:48:22.743 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.743+0000 7f3e8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.756 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.755+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589292457 0 0) 0x7f3ea4137f10 con 0x7f3ea41430c0 2024-03-25T19:48:22.756 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.755+0000 7f3e8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.756 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.755+0000 7f3e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.756 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.755+0000 7f3e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.756 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.755+0000 7f3e8b7fe640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:48:22.756 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.755+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e90004eb0 con 0x7f3ea41430c0 2024-03-25T19:48:22.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.757+0000 7f3e8b7fe640 10 client.?.objecter ms_handle_connect 0x7f3ea40a98f0 2024-03-25T19:48:22.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.757+0000 7f3e8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.762 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.762+0000 7f3e8b7fe640 10 client.?.objecter ms_handle_connect 0x7f3ea413f710 2024-03-25T19:48:22.762 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.762+0000 7f3e8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.785+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3979761205 0 0) 0x7f3e90004eb0 con 0x7f3ea41430c0 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.785+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ea4137f10 con 0x7f3ea41430c0 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.785+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3e9c001fc0 con 0x7f3ea41430c0 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.785+0000 7f3e8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.785+0000 7f3e8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.785+0000 7f3e8b7fe640 10 monclient(hunting): dump: 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.785+0000 7f3e8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:22.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1710938193 0 0) 0x7f3ea4137f10 con 0x7f3ea41430c0 2024-03-25T19:48:22.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 >> v1:172.21.15.69:6790/0 conn(0x7f3ea413f710 legacy=0x7f3ea41db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 >> v1:172.21.15.169:6789/0 conn(0x7f3ea40a98f0 legacy=0x7f3ea4137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:22.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 1 monclient: found mon.a 2024-03-25T19:48:22.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:22.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea41373e0 con 0x7f3ea41430c0 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3e8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 5 monclient: authenticate success, global_id 4575 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 10 monclient: _renew_subs 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea41358e0 con 0x7f3ea41430c0 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 10 monclient: _renew_subs 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea4136150 con 0x7f3ea41430c0 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.845+0000 7f3ea960f640 1 librados: init done 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.846+0000 7f3ea960f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:22.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.846+0000 7f3ea960f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:22.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.846+0000 7f3ea960f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:48:22.848 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:22.846+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea40627b0 con 0x7f3ea41430c0 2024-03-25T19:48:23.146 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.146+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e9c003c20 con 0x7f3ea41430c0 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.146+0000 7f3e8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.146+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3e9c004030 con 0x7f3ea41430c0 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.146+0000 7f3e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.146+0000 7f3e8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.146+0000 7f3e8b7fe640 10 monclient: dump: 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:23.147 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:23.148 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:23.148 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:23.148 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:23.148 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:23.148 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.146+0000 7f3ea17fa640 4 set_mon_vals no callback set 2024-03-25T19:48:23.148 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.147+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f3e9c004250 con 0x7f3ea41430c0 2024-03-25T19:48:23.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.148+0000 7f3ea2ffd640 10 monclient: get_auth_request con 0x7f3e9006dd00 auth_method 0 2024-03-25T19:48:23.149 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.148+0000 7f3e8b7fe640 10 client.4575.objecter ms_handle_connect 0x7f3e9006dd00 2024-03-25T19:48:23.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.155+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f3e9c054c60 con 0x7f3ea41430c0 2024-03-25T19:48:23.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.155+0000 7f3e8b7fe640 10 client.4575.objecter ms_dispatch 0x7f3ea4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:23.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.155+0000 7f3e8b7fe640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:23.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.155+0000 7f3e8b7fe640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:23.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.155+0000 7f3e8b7fe640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T19:48:23.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.155+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e9c050770 con 0x7f3ea41430c0 2024-03-25T19:48:23.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.155+0000 7f3e8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:23.156 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.155+0000 7f3e8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:23.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.394+0000 7f3ea960f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.394+0000 7f3ea960f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.394+0000 7f3ea960f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:48:23.395 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.394+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ea4135a70 con 0x7f3ea41430c0 2024-03-25T19:48:23.436 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.436+0000 7f3e8b7fe640 1 -- 172.21.15.69:0/1239831788 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f3e9c0526a0 con 0x7f3ea41430c0 2024-03-25T19:48:23.436 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.436+0000 7f3e8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.436 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.436+0000 7f3e8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:23.436 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:48:23.437 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:23.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.438+0000 7f3ea960f640 10 librados: watch_flush enter 2024-03-25T19:48:23.439 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 10 librados: watch_flush exit 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 10 client.4575.objecter successfully canceled tick 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 >> v1:172.21.15.169:6800/3072548259 conn(0x7f3e9006dd00 legacy=0x7f3e900701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 10 monclient: shutdown 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 >> v1:172.21.15.69:6789/0 conn(0x7f3ea41430c0 legacy=0x7f3ea41db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 shutdown_connections 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 >> 172.21.15.69:0/1239831788 conn(0x7f3ea40b7ea0 msgr2=0x7f3ea414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 shutdown_connections 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 1 -- 172.21.15.69:0/1239831788 wait complete. 2024-03-25T19:48:23.440 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:23.439+0000 7f3ea960f640 1 librados: shutdown 2024-03-25T19:48:23.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:28.318 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:06.282 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.324 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:06.288 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.358 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:06.322 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.371 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:06.325 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.381 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:06.335 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.386 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:06.350 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.396 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:06.360 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.416 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:06.370 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.428 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:06.392 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:30.258 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:30.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.368+0000 7f1a639a3640 10 monclient: get_monmap_and_config 2024-03-25T19:48:30.367 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.368+0000 7f1a639a3640 10 monclient: build_initial_monmap 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.368+0000 7f1a639a3640 10 monclient: monmap: 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:48:30.368655+0000 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:48:30.368655+0000 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:48:30.368 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:30.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.369+0000 7f1a639a3640 1 Processor -- start 2024-03-25T19:48:30.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.369+0000 7f1a639a3640 1 -- start start 2024-03-25T19:48:30.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.369+0000 7f1a639a3640 10 monclient: init 2024-03-25T19:48:30.369 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5c0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 10 monclient(hunting): picked mon.noname-c con 0x7f1a5c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 10 monclient(hunting): picked mon.noname-a con 0x7f1a5c1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c142520 con 0x7f1a5c1430c0 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0b28a0 con 0x7f1a5c13f710 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0b2a30 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:30.370 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.370+0000 7f1a639a3640 10 monclient(hunting): authenticate will time out at 5322.638027s 2024-03-25T19:48:30.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.382+0000 7f1a61f40640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1a5c1430c0 0x7f1a5c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60688/0 (socket says 172.21.15.69:60688) 2024-03-25T19:48:30.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.382+0000 7f1a61f40640 1 -- 172.21.15.69:0/1568227818 learned_addr learned my addr 172.21.15.69:0/1568227818 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:30.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.393+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367736723 0 0) 0x7f1a5c0b2a30 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.393+0000 7f1a53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.393+0000 7f1a53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.393+0000 7f1a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.393+0000 7f1a53fff640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:48:30.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.393+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a48003520 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.398+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764576222 0 0) 0x7f1a5c0b28a0 con 0x7f1a5c13f710 2024-03-25T19:48:30.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.398+0000 7f1a53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.398+0000 7f1a53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.398+0000 7f1a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.398+0000 7f1a53fff640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:48:30.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.398+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a5c0b2a30 con 0x7f1a5c13f710 2024-03-25T19:48:30.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.400+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097864783 0 0) 0x7f1a5c142520 con 0x7f1a5c1430c0 2024-03-25T19:48:30.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.400+0000 7f1a53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.400+0000 7f1a53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.400+0000 7f1a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.400+0000 7f1a53fff640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:48:30.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.400+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a5c0b28a0 con 0x7f1a5c1430c0 2024-03-25T19:48:30.408 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.408+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1178415224 0 0) 0x7f1a48003520 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.408 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 >> v1:172.21.15.169:6789/0 conn(0x7f1a5c13f710 legacy=0x7f1a5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.408 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 >> v1:172.21.15.69:6789/0 conn(0x7f1a5c1430c0 legacy=0x7f1a5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.408 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 1 monclient: found mon.noname-b 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0b44c0 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 10 monclient: _finish_auth 0 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1a58001690 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a639a3640 5 monclient: authenticate success, global_id 4556 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a639a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a53fff640 10 monclient: dump: 2024-03-25T19:48:30.409 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:30.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.409+0000 7f1a639a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:30.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.425+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a580017f0 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.425+0000 7f1a53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:30.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.425+0000 7f1a53fff640 1 -- 172.21.15.69:0/1568227818 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1a58001c00 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.425+0000 7f1a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.425+0000 7f1a53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:30.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.425+0000 7f1a53fff640 10 monclient: dump: 2024-03-25T19:48:30.424 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.425+0000 7f1a639a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:30.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.425+0000 7f1a639a3640 4 set_mon_vals no callback set 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.426+0000 7f1a639a3640 10 monclient: shutdown 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.426+0000 7f1a639a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.426+0000 7f1a639a3640 1 -- 172.21.15.69:0/1568227818 >> v1:172.21.15.69:6790/0 conn(0x7f1a5c0a98f0 legacy=0x7f1a5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.426+0000 7f1a639a3640 1 -- 172.21.15.69:0/1568227818 shutdown_connections 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.426+0000 7f1a639a3640 1 -- 172.21.15.69:0/1568227818 >> 172.21.15.69:0/1568227818 conn(0x7f1a5c0b7ea0 msgr2=0x7f1a5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.426+0000 7f1a639a3640 1 -- 172.21.15.69:0/1568227818 shutdown_connections 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.426+0000 7f1a639a3640 1 -- 172.21.15.69:0/1568227818 wait complete. 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 10 monclient: build_initial_monmap 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 10 monclient: monmap: 2024-03-25T19:48:30.427 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 1 librados: starting msgr at 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 1 librados: starting objecter 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 1 Processor -- start 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 1 -- start start 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 1 librados: setting wanted keys 2024-03-25T19:48:30.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 1 librados: calling monclient init 2024-03-25T19:48:30.429 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.427+0000 7f1a639a3640 10 monclient: init 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.428+0000 7f1a639a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.428+0000 7f1a639a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.428+0000 7f1a639a3640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.428+0000 7f1a639a3640 10 monclient(hunting): picked mon.noname-a con 0x7f1a5c13f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.428+0000 7f1a639a3640 10 monclient(hunting): picked mon.noname-c con 0x7f1a5c1430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.429+0000 7f1a639a3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c1db910 con 0x7f1a5c13f710 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.429+0000 7f1a639a3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c071f90 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.429+0000 7f1a639a3640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c1dc540 con 0x7f1a5c1430c0 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.429+0000 7f1a639a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:30.430 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.429+0000 7f1a639a3640 10 monclient(hunting): authenticate will time out at 5322.696768s 2024-03-25T19:48:30.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.444+0000 7f1a60f3e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1a5c13f710 0x7f1a5c070b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60726/0 (socket says 172.21.15.69:60726) 2024-03-25T19:48:30.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.444+0000 7f1a60f3e640 1 -- 172.21.15.69:0/1969408543 learned_addr learned my addr 172.21.15.69:0/1969408543 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:30.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.446+0000 7f1a51ffb640 10 client.?.objecter ms_handle_connect 0x7f1a5c13f710 2024-03-25T19:48:30.445 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.446+0000 7f1a51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.448+0000 7f1a51ffb640 10 client.?.objecter ms_handle_connect 0x7f1a5c1430c0 2024-03-25T19:48:30.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.448+0000 7f1a51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.453+0000 7f1a51ffb640 10 client.?.objecter ms_handle_connect 0x7f1a5c0a98f0 2024-03-25T19:48:30.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.453+0000 7f1a51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.456+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667594924 0 0) 0x7f1a5c1dc540 con 0x7f1a5c1430c0 2024-03-25T19:48:30.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.456+0000 7f1a51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.456+0000 7f1a51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.456+0000 7f1a51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.456+0000 7f1a51ffb640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:48:30.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.456+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a40004eb0 con 0x7f1a5c1430c0 2024-03-25T19:48:30.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.465+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292689030 0 0) 0x7f1a5c071f90 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.464 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.465+0000 7f1a51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.465+0000 7f1a51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.465+0000 7f1a51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.465+0000 7f1a51ffb640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:48:30.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.465+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a5c1dc540 con 0x7f1a5c0a98f0 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.466+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 896822218 0 0) 0x7f1a40004eb0 con 0x7f1a5c1430c0 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.466+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a5c071f90 con 0x7f1a5c1430c0 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.466+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1a54001bd0 con 0x7f1a5c1430c0 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.466+0000 7f1a51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.466+0000 7f1a51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.466+0000 7f1a51ffb640 10 monclient(hunting): dump: 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:30.467 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:30.467 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:30.467 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:30.467 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.466+0000 7f1a51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:30.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1019185525 0 0) 0x7f1a5c071f90 con 0x7f1a5c1430c0 2024-03-25T19:48:30.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 >> v1:172.21.15.169:6789/0 conn(0x7f1a5c0a98f0 legacy=0x7f1a5c1dbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 >> v1:172.21.15.69:6789/0 conn(0x7f1a5c13f710 legacy=0x7f1a5c070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:30.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 1 monclient: found mon.c 2024-03-25T19:48:30.470 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c1dc6d0 con 0x7f1a5c1430c0 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a639a3640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a639a3640 10 monclient: _renew_subs 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a639a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.470+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c06dea0 con 0x7f1a5c1430c0 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.471+0000 7f1a639a3640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.471+0000 7f1a639a3640 10 monclient: _renew_subs 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.471+0000 7f1a639a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.471+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c071440 con 0x7f1a5c1430c0 2024-03-25T19:48:30.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.471+0000 7f1a639a3640 1 librados: init done 2024-03-25T19:48:30.472 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.472+0000 7f1a639a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.472 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.472+0000 7f1a639a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.472 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.472+0000 7f1a639a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:30.472 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.472+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c0627b0 con 0x7f1a5c1430c0 2024-03-25T19:48:30.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.782+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a54001fe0 con 0x7f1a5c1430c0 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.782+0000 7f1a51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.782+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f1a54004d00 con 0x7f1a5c1430c0 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.782+0000 7f1a51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.782+0000 7f1a51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.782+0000 7f1a51ffb640 10 monclient: dump: 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:30.782 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:30.783 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:30.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.782+0000 7f1a537fe640 4 set_mon_vals no callback set 2024-03-25T19:48:30.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.783+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f1a54004f20 con 0x7f1a5c1430c0 2024-03-25T19:48:30.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.783+0000 7f1a6173f640 10 monclient: get_auth_request con 0x7f1a4006dd10 auth_method 0 2024-03-25T19:48:30.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.783+0000 7f1a51ffb640 10 client.4559.objecter ms_handle_connect 0x7f1a4006dd10 2024-03-25T19:48:30.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.788+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f1a54054a80 con 0x7f1a5c1430c0 2024-03-25T19:48:30.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.788+0000 7f1a51ffb640 10 client.4559.objecter ms_dispatch 0x7f1a5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:30.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.788+0000 7f1a51ffb640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:30.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.788+0000 7f1a51ffb640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:30.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.788+0000 7f1a51ffb640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-25T19:48:30.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.788+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a54050590 con 0x7f1a5c1430c0 2024-03-25T19:48:30.787 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.788+0000 7f1a51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.788 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:30.788+0000 7f1a51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:31.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.007+0000 7f1a639a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:31.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.007+0000 7f1a639a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:31.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.007+0000 7f1a639a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:31.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.007+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a5c1dbaa0 con 0x7f1a5c1430c0 2024-03-25T19:48:31.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.037+0000 7f1a51ffb640 1 -- 172.21.15.69:0/1969408543 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f1a540524c0 con 0x7f1a5c1430c0 2024-03-25T19:48:31.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.037+0000 7f1a51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:31.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.037+0000 7f1a51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:31.037 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:48:31.037 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:31.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 10 librados: watch_flush enter 2024-03-25T19:48:31.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 10 librados: watch_flush exit 2024-03-25T19:48:31.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-25T19:48:31.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 10 client.4559.objecter successfully canceled tick 2024-03-25T19:48:31.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 >> v1:172.21.15.169:6800/3072548259 conn(0x7f1a4006dd10 legacy=0x7f1a400701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:31.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 10 monclient: shutdown 2024-03-25T19:48:31.039 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:31.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 >> v1:172.21.15.69:6790/0 conn(0x7f1a5c1430c0 legacy=0x7f1a5c1e6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:31.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 shutdown_connections 2024-03-25T19:48:31.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.039+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 >> 172.21.15.69:0/1969408543 conn(0x7f1a5c0b7ea0 msgr2=0x7f1a5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:31.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.040+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 shutdown_connections 2024-03-25T19:48:31.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.040+0000 7f1a639a3640 1 -- 172.21.15.69:0/1969408543 wait complete. 2024-03-25T19:48:31.040 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:31.040+0000 7f1a639a3640 1 librados: shutdown 2024-03-25T19:48:31.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:37.858 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.973+0000 7f93e1d6c640 10 monclient: get_monmap_and_config 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.973+0000 7f93e1d6c640 10 monclient: build_initial_monmap 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.973+0000 7f93e1d6c640 10 monclient: monmap: 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:48:37.974897+0000 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:48:37.974897+0000 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:48:37.974 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:37.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.974+0000 7f93e1d6c640 1 Processor -- start 2024-03-25T19:48:37.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.974+0000 7f93e1d6c640 1 -- start start 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.974+0000 7f93e1d6c640 10 monclient: init 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 10 monclient(hunting): picked mon.noname-b con 0x7f93dc0b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 10 monclient(hunting): picked mon.noname-a con 0x7f93dc0b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 10 monclient(hunting): picked mon.noname-c con 0x7f93dc14be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc0af940 con 0x7f93dc0b1730 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc0b0a70 con 0x7f93dc14be90 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc0b0d50 con 0x7f93dc0b4d60 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:37.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.975+0000 7f93e1d6c640 10 monclient(hunting): authenticate will time out at 5330.244240s 2024-03-25T19:48:37.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.990+0000 7f93dbfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f93dc14be90 0x7f93dc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:35392/0 (socket says 172.21.15.69:35392) 2024-03-25T19:48:37.990 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:37.990+0000 7f93dbfff640 1 -- 172.21.15.69:0/3664474341 learned_addr learned my addr 172.21.15.69:0/3664474341 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:38.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.005+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640788724 0 0) 0x7f93dc0b0a70 con 0x7f93dc14be90 2024-03-25T19:48:38.005 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.005+0000 7f93da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:38.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.005+0000 7f93da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:38.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.005+0000 7f93da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:38.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.005+0000 7f93da7fc640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:48:38.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.005+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c4003520 con 0x7f93dc14be90 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.011+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295660148 0 0) 0x7f93dc0af940 con 0x7f93dc0b1730 2024-03-25T19:48:38.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.011+0000 7f93da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:38.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.011+0000 7f93da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:38.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.011+0000 7f93da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:38.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.011+0000 7f93da7fc640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:48:38.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.011+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93dc0b0a70 con 0x7f93dc0b1730 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2302874980 0 0) 0x7f93c4003520 con 0x7f93dc14be90 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 >> v1:172.21.15.69:6790/0 conn(0x7f93dc0b4d60 legacy=0x7f93dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 >> v1:172.21.15.69:6789/0 conn(0x7f93dc0b1730 legacy=0x7f93dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 1 monclient: found mon.noname-c 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc0b1030 con 0x7f93dc14be90 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f93d4001a10 con 0x7f93dc14be90 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:38.028 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.027+0000 7f93da7fc640 10 monclient: dump: 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.028+0000 7f93e1d6c640 5 monclient: authenticate success, global_id 4552 2024-03-25T19:48:38.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.028+0000 7f93e1d6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.053+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93d4001b70 con 0x7f93dc14be90 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.053+0000 7f93da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.053+0000 7f93da7fc640 1 -- 172.21.15.69:0/3664474341 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f93d4001f80 con 0x7f93dc14be90 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.053+0000 7f93da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.053+0000 7f93da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.053+0000 7f93da7fc640 10 monclient: dump: 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:38.053 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.053+0000 7f93e1d6c640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:38.054 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.053+0000 7f93e1d6c640 4 set_mon_vals no callback set 2024-03-25T19:48:38.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.054+0000 7f93e1d6c640 10 monclient: shutdown 2024-03-25T19:48:38.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.054+0000 7f93e1d6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:38.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.054+0000 7f93e1d6c640 1 -- 172.21.15.69:0/3664474341 >> v1:172.21.15.169:6789/0 conn(0x7f93dc14be90 legacy=0x7f93dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.054+0000 7f93e1d6c640 1 -- 172.21.15.69:0/3664474341 shutdown_connections 2024-03-25T19:48:38.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.054+0000 7f93e1d6c640 1 -- 172.21.15.69:0/3664474341 >> 172.21.15.69:0/3664474341 conn(0x7f93dc0617a0 msgr2=0x7f93dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:38.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.054+0000 7f93e1d6c640 1 -- 172.21.15.69:0/3664474341 shutdown_connections 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.054+0000 7f93e1d6c640 1 -- 172.21.15.69:0/3664474341 wait complete. 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.055+0000 7f93e1d6c640 10 monclient: build_initial_monmap 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.055+0000 7f93e1d6c640 10 monclient: monmap: 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:48:38.056 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.055+0000 7f93e1d6c640 1 librados: starting msgr at 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.055+0000 7f93e1d6c640 1 librados: starting objecter 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.056+0000 7f93e1d6c640 1 Processor -- start 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.056+0000 7f93e1d6c640 1 -- start start 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.056+0000 7f93e1d6c640 1 librados: setting wanted keys 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.056+0000 7f93e1d6c640 1 librados: calling monclient init 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.056+0000 7f93e1d6c640 10 monclient: init 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 10 monclient(hunting): picked mon.noname-c con 0x7f93dc0b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 10 monclient(hunting): picked mon.noname-a con 0x7f93dc0b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:38.058 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 10 monclient(hunting): picked mon.noname-b con 0x7f93dc14be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:38.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc138b40 con 0x7f93dc0b4d60 2024-03-25T19:48:38.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc145240 con 0x7f93dc14be90 2024-03-25T19:48:38.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc145520 con 0x7f93dc0b1730 2024-03-25T19:48:38.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:38.059 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.057+0000 7f93e1d6c640 10 monclient(hunting): authenticate will time out at 5330.325908s 2024-03-25T19:48:38.071 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.071+0000 7f93daffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f93dc0b4d60 0x7f93dc1d7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42430/0 (socket says 172.21.15.69:42430) 2024-03-25T19:48:38.071 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.071+0000 7f93daffd640 1 -- 172.21.15.69:0/36618693 learned_addr learned my addr 172.21.15.69:0/36618693 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:38.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.072+0000 7f93c3fff640 10 client.?.objecter ms_handle_connect 0x7f93dc0b4d60 2024-03-25T19:48:38.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.072+0000 7f93c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:38.073 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.074+0000 7f93c3fff640 10 client.?.objecter ms_handle_connect 0x7f93dc14be90 2024-03-25T19:48:38.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.074+0000 7f93c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:38.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.076+0000 7f93c3fff640 10 client.?.objecter ms_handle_connect 0x7f93dc0b1730 2024-03-25T19:48:38.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.076+0000 7f93c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.080+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612109496 0 0) 0x7f93dc145520 con 0x7f93dc0b1730 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.080+0000 7f93c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.080+0000 7f93c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.080+0000 7f93c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.080+0000 7f93c3fff640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.080+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c8004eb0 con 0x7f93dc0b1730 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.081+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302067841 0 0) 0x7f93dc145240 con 0x7f93dc14be90 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.081+0000 7f93c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.081+0000 7f93c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:38.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.081+0000 7f93c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:38.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.081+0000 7f93c3fff640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:48:38.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.081+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93dc145520 con 0x7f93dc14be90 2024-03-25T19:48:38.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.089+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1889381521 0 0) 0x7f93c8004eb0 con 0x7f93dc0b1730 2024-03-25T19:48:38.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.089+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93dc145240 con 0x7f93dc0b1730 2024-03-25T19:48:38.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.089+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f93cc001930 con 0x7f93dc0b1730 2024-03-25T19:48:38.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.089+0000 7f93c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:38.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.089+0000 7f93c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:38.089 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.089+0000 7f93c3fff640 10 monclient(hunting): dump: 2024-03-25T19:48:38.089 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:38.089 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:38.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.089+0000 7f93c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:38.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.092+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377278118 0 0) 0x7f93dc138b40 con 0x7f93dc0b4d60 2024-03-25T19:48:38.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.092+0000 7f93c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:38.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.092+0000 7f93c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:38.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.092+0000 7f93c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:38.093 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.092+0000 7f93c3fff640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:48:38.093 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.092+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c8004eb0 con 0x7f93dc0b4d60 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4196719886 0 0) 0x7f93dc145240 con 0x7f93dc0b1730 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 >> v1:172.21.15.169:6789/0 conn(0x7f93dc14be90 legacy=0x7f93dc1e24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 >> v1:172.21.15.69:6789/0 conn(0x7f93dc0b4d60 legacy=0x7f93dc1d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 1 monclient: found mon.c 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc1405f0 con 0x7f93dc0b1730 2024-03-25T19:48:38.094 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 10 monclient: _finish_auth 0 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 5 monclient: authenticate success, global_id 4565 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 10 monclient: _renew_subs 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc1408a0 con 0x7f93dc0b1730 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 10 monclient: _renew_subs 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc1d8270 con 0x7f93dc0b1730 2024-03-25T19:48:38.095 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.093+0000 7f93e1d6c640 1 librados: init done 2024-03-25T19:48:38.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.095+0000 7f93e1d6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:38.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.095+0000 7f93e1d6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:38.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.095+0000 7f93e1d6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:38.096 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.095+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc1419c0 con 0x7f93dc0b1730 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.402+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93cc003610 con 0x7f93dc0b1730 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.402+0000 7f93c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.402+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f93cc0039a0 con 0x7f93dc0b1730 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.402+0000 7f93c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.402+0000 7f93c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.402+0000 7f93c3fff640 10 monclient: dump: 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:38.403 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:38.404 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:38.404 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:38.404 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:38.404 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:38.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.402+0000 7f93d9ffb640 4 set_mon_vals no callback set 2024-03-25T19:48:38.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.403+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f93cc003bc0 con 0x7f93dc0b1730 2024-03-25T19:48:38.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.404+0000 7f93daffd640 10 monclient: get_auth_request con 0x7f93c806de00 auth_method 0 2024-03-25T19:48:38.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.404+0000 7f93c3fff640 10 client.4565.objecter ms_handle_connect 0x7f93c806de00 2024-03-25T19:48:38.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.409+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f93cc053620 con 0x7f93dc0b1730 2024-03-25T19:48:38.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.409+0000 7f93c3fff640 10 client.4565.objecter ms_dispatch 0x7f93dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:38.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.409+0000 7f93c3fff640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:38.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.409+0000 7f93c3fff640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:38.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.409+0000 7f93c3fff640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-25T19:48:38.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.409+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93cc04f130 con 0x7f93dc0b1730 2024-03-25T19:48:38.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.409+0000 7f93c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:38.409 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.409+0000 7f93c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:38.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.660+0000 7f93e1d6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.660+0000 7f93e1d6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.660+0000 7f93e1d6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:38.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.660+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93dc0627b0 con 0x7f93dc0b1730 2024-03-25T19:48:38.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.780+0000 7f93c3fff640 1 -- 172.21.15.69:0/36618693 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f93cc051060 con 0x7f93dc0b1730 2024-03-25T19:48:38.780 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.780+0000 7f93c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.781 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.780+0000 7f93c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:38.781 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:48:38.781 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:38.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 10 librados: watch_flush enter 2024-03-25T19:48:38.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 10 librados: watch_flush exit 2024-03-25T19:48:38.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-25T19:48:38.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 10 client.4565.objecter successfully canceled tick 2024-03-25T19:48:38.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 >> v1:172.21.15.169:6800/3072548259 conn(0x7f93c806de00 legacy=0x7f93c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 10 monclient: shutdown 2024-03-25T19:48:38.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:38.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 >> v1:172.21.15.69:6790/0 conn(0x7f93dc0b1730 legacy=0x7f93dc138430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 shutdown_connections 2024-03-25T19:48:38.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.782+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 >> 172.21.15.69:0/36618693 conn(0x7f93dc0617a0 msgr2=0x7f93dc0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:38.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.783+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 shutdown_connections 2024-03-25T19:48:38.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.783+0000 7f93e1d6c640 1 -- 172.21.15.69:0/36618693 wait complete. 2024-03-25T19:48:38.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:38.783+0000 7f93e1d6c640 1 librados: shutdown 2024-03-25T19:48:38.795 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:45.602 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.703+0000 7f04aa67b640 10 monclient: get_monmap_and_config 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.703+0000 7f04aa67b640 10 monclient: build_initial_monmap 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.703+0000 7f04aa67b640 10 monclient: monmap: 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:48:45.704888+0000 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:48:45.704888+0000 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:48:45.704 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:48:45.705 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:45.705 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.704+0000 7f04aa67b640 1 Processor -- start 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.704+0000 7f04aa67b640 1 -- start start 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.704+0000 7f04aa67b640 10 monclient: init 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 10 monclient(hunting): picked mon.noname-c con 0x7f04a40b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 10 monclient(hunting): picked mon.noname-a con 0x7f04a40b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 10 monclient(hunting): picked mon.noname-b con 0x7f04a414be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a40af940 con 0x7f04a40b1730 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a40b0a70 con 0x7f04a40b4d60 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a40b0d50 con 0x7f04a414be90 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:45.706 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.705+0000 7f04aa67b640 10 monclient(hunting): authenticate will time out at 5337.974030s 2024-03-25T19:48:45.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.716+0000 7f04a37fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f04a40b1730 0x7f04a40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45700/0 (socket says 172.21.15.69:45700) 2024-03-25T19:48:45.716 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.716+0000 7f04a37fe640 1 -- 172.21.15.69:0/3704381685 learned_addr learned my addr 172.21.15.69:0/3704381685 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:45.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.736+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602467717 0 0) 0x7f04a40b0d50 con 0x7f04a414be90 2024-03-25T19:48:45.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.736+0000 7f04a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.736+0000 7f04a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.736+0000 7f04a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.736+0000 7f04a2ffd640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:48:45.736 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.736+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003520 con 0x7f04a414be90 2024-03-25T19:48:45.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340232472 0 0) 0x7f04a40af940 con 0x7f04a40b1730 2024-03-25T19:48:45.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:48:45.737 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a40b0d50 con 0x7f04a40b1730 2024-03-25T19:48:45.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543538579 0 0) 0x7f04a40b0a70 con 0x7f04a40b4d60 2024-03-25T19:48:45.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:48:45.738 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.737+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a40af940 con 0x7f04a40b4d60 2024-03-25T19:48:45.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3356998201 0 0) 0x7f04a40af940 con 0x7f04a40b4d60 2024-03-25T19:48:45.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 >> v1:172.21.15.69:6790/0 conn(0x7f04a414be90 legacy=0x7f04a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 >> v1:172.21.15.69:6789/0 conn(0x7f04a40b1730 legacy=0x7f04a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:45.745 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 1 monclient: found mon.noname-c 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a40b1030 con 0x7f04a40b4d60 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f04940015b0 con 0x7f04a40b4d60 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04a2ffd640 10 monclient: dump: 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:45.746 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04aa67b640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:48:45.747 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.745+0000 7f04aa67b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:45.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.752+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0494001710 con 0x7f04a40b4d60 2024-03-25T19:48:45.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.752+0000 7f04a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:45.752 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.752+0000 7f04aa67b640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:45.753 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.752+0000 7f04aa67b640 4 set_mon_vals no callback set 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04a2ffd640 1 -- 172.21.15.69:0/3704381685 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0494001b20 con 0x7f04a40b4d60 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04a2ffd640 10 monclient: dump: 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:45.754 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04aa67b640 10 monclient: shutdown 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04aa67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04aa67b640 1 -- 172.21.15.69:0/3704381685 >> v1:172.21.15.169:6789/0 conn(0x7f04a40b4d60 legacy=0x7f04a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04aa67b640 1 -- 172.21.15.69:0/3704381685 shutdown_connections 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04aa67b640 1 -- 172.21.15.69:0/3704381685 >> 172.21.15.69:0/3704381685 conn(0x7f04a40617a0 msgr2=0x7f04a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04aa67b640 1 -- 172.21.15.69:0/3704381685 shutdown_connections 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.753+0000 7f04aa67b640 1 -- 172.21.15.69:0/3704381685 wait complete. 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 10 monclient: build_initial_monmap 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 10 monclient: monmap: 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:45.755 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:48:45.756 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:48:45.756 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:45.756 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:48:45.756 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:45.756 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:48:45.756 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:48:45.756 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:45.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 1 librados: starting msgr at 2024-03-25T19:48:45.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 1 librados: starting objecter 2024-03-25T19:48:45.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 1 Processor -- start 2024-03-25T19:48:45.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 1 -- start start 2024-03-25T19:48:45.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 1 librados: setting wanted keys 2024-03-25T19:48:45.757 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 1 librados: calling monclient init 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.754+0000 7f04aa67b640 10 monclient: init 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 10 monclient(hunting): picked mon.noname-a con 0x7f04a40b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 10 monclient(hunting): picked mon.noname-b con 0x7f04a40b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 10 monclient(hunting): picked mon.noname-c con 0x7f04a414be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a4142ee0 con 0x7f04a40b1730 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a41421e0 con 0x7f04a40b4d60 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a413e930 con 0x7f04a414be90 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:45.758 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.755+0000 7f04aa67b640 10 monclient(hunting): authenticate will time out at 5338.024430s 2024-03-25T19:48:45.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.761+0000 7f04a8c18640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f04a414be90 0x7f04a41e70b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:49150/0 (socket says 172.21.15.69:49150) 2024-03-25T19:48:45.761 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.761+0000 7f04a8c18640 1 -- 172.21.15.69:0/2953968047 learned_addr learned my addr 172.21.15.69:0/2953968047 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:45.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.767+0000 7f04a0ff9640 10 client.?.objecter ms_handle_connect 0x7f04a414be90 2024-03-25T19:48:45.767 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.767+0000 7f04a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.769+0000 7f04a0ff9640 10 client.?.objecter ms_handle_connect 0x7f04a40b1730 2024-03-25T19:48:45.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.769+0000 7f04a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.770 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.770+0000 7f04a0ff9640 10 client.?.objecter ms_handle_connect 0x7f04a40b4d60 2024-03-25T19:48:45.770 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.770+0000 7f04a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.773+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488590859 0 0) 0x7f04a413e930 con 0x7f04a414be90 2024-03-25T19:48:45.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.773+0000 7f04a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.773+0000 7f04a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.773+0000 7f04a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.773+0000 7f04a0ff9640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:48:45.773 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.773+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0490004eb0 con 0x7f04a414be90 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.782+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819568245 0 0) 0x7f0490004eb0 con 0x7f04a414be90 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.783+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a413e930 con 0x7f04a414be90 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.783+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f049c001c30 con 0x7f04a414be90 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.783+0000 7f04a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.783+0000 7f04a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.783+0000 7f04a0ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:45.783 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:45.784 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.784 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:45.784 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:45.784 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:45.784 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:45.784 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:45.784 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.783+0000 7f04a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:45.785 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.785+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529887637 0 0) 0x7f04a41421e0 con 0x7f04a40b4d60 2024-03-25T19:48:45.785 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.785+0000 7f04a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.785+0000 7f04a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.785+0000 7f04a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.785+0000 7f04a0ff9640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:48:45.786 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.785+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0490004eb0 con 0x7f04a40b4d60 2024-03-25T19:48:45.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.792+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760349989 0 0) 0x7f04a4142ee0 con 0x7f04a40b1730 2024-03-25T19:48:45.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.792+0000 7f04a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.792+0000 7f04a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.792+0000 7f04a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.792+0000 7f04a0ff9640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:48:45.792 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.792+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a41421e0 con 0x7f04a40b1730 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.798+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3389723603 0 0) 0x7f0490004eb0 con 0x7f04a40b4d60 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.798+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a4142ee0 con 0x7f04a40b4d60 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.798+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f0498001930 con 0x7f04a40b4d60 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.798+0000 7f04a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.798+0000 7f04a0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.798+0000 7f04a0ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:45.798 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:45.799 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:45.799 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.799 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:45.799 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:45.799 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:45.799 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:45.799 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:45.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 628035687 0 0) 0x7f04a413e930 con 0x7f04a414be90 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 >> v1:172.21.15.169:6789/0 conn(0x7f04a40b4d60 legacy=0x7f04a413e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 >> v1:172.21.15.69:6789/0 conn(0x7f04a40b1730 legacy=0x7f04a41418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 1 monclient: found mon.c 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a4138ea0 con 0x7f04a414be90 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 5 monclient: authenticate success, global_id 4571 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 10 monclient: _renew_subs 2024-03-25T19:48:45.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a4143130 con 0x7f04a414be90 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 10 monclient: _renew_subs 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a4140380 con 0x7f04a414be90 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.810+0000 7f04aa67b640 1 librados: init done 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.811+0000 7f04aa67b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.811+0000 7f04aa67b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.811+0000 7f04aa67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:45.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:45.811+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a40627b0 con 0x7f04a414be90 2024-03-25T19:48:46.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.108+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f049c003990 con 0x7f04a414be90 2024-03-25T19:48:46.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.108+0000 7f04a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.108+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f049c003d20 con 0x7f04a414be90 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.108+0000 7f04a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.108+0000 7f04a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.108+0000 7f04a0ff9640 10 monclient: dump: 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:46.109 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:46.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.108+0000 7f04a27fc640 4 set_mon_vals no callback set 2024-03-25T19:48:46.110 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.109+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f049c003f40 con 0x7f04a414be90 2024-03-25T19:48:46.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.109+0000 7f04a3fff640 10 monclient: get_auth_request con 0x7f049006dfd0 auth_method 0 2024-03-25T19:48:46.111 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.110+0000 7f04a0ff9640 10 client.4571.objecter ms_handle_connect 0x7f049006dfd0 2024-03-25T19:48:46.116 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.115+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f049c0549d0 con 0x7f04a414be90 2024-03-25T19:48:46.116 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.115+0000 7f04a0ff9640 10 client.4571.objecter ms_dispatch 0x7f04a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:46.116 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.115+0000 7f04a0ff9640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:46.116 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.115+0000 7f04a0ff9640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:46.116 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.115+0000 7f04a0ff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-25T19:48:46.116 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.115+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f049c0504e0 con 0x7f04a414be90 2024-03-25T19:48:46.116 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.115+0000 7f04a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:46.116 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.115+0000 7f04a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:46.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.371+0000 7f04aa67b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.371+0000 7f04aa67b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.371 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.371+0000 7f04aa67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:46.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.371+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04a4072620 con 0x7f04a414be90 2024-03-25T19:48:46.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.399+0000 7f04a0ff9640 1 -- 172.21.15.69:0/2953968047 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f049c052410 con 0x7f04a414be90 2024-03-25T19:48:46.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.399+0000 7f04a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.399+0000 7f04a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:46.400 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:48:46.400 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 10 librados: watch_flush enter 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 10 librados: watch_flush exit 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 10 client.4571.objecter successfully canceled tick 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 >> v1:172.21.15.169:6800/3072548259 conn(0x7f049006dfd0 legacy=0x7f0490070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 10 monclient: shutdown 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 >> v1:172.21.15.69:6790/0 conn(0x7f04a414be90 legacy=0x7f04a41e70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 shutdown_connections 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 >> 172.21.15.69:0/2953968047 conn(0x7f04a40617a0 msgr2=0x7f04a4143900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 shutdown_connections 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 1 -- 172.21.15.69:0/2953968047 wait complete. 2024-03-25T19:48:46.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:46.402+0000 7f04aa67b640 1 librados: shutdown 2024-03-25T19:48:46.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:48.576 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:26.540 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.586 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:26.550 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.623 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:26.587 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.641 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:26.605 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.649 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:26.613 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.654 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:26.608 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.659 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:26.613 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.686 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:26.650 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.699 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:26.653 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:53.220 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.334+0000 7f9287bf4640 10 monclient: get_monmap_and_config 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.335+0000 7f9287bf4640 10 monclient: build_initial_monmap 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.335+0000 7f9287bf4640 10 monclient: monmap: 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:48:53.336234+0000 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:48:53.336234+0000 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.335+0000 7f9287bf4640 1 Processor -- start 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 1 -- start start 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 10 monclient: init 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f92800b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f92800b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f928014be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92800af940 con 0x7f92800b4d60 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92800b0a70 con 0x7f928014be90 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92800b0d50 con 0x7f92800b1730 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:53.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.336+0000 7f9287bf4640 10 monclient(hunting): authenticate will time out at 5345.605381s 2024-03-25T19:48:53.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.348+0000 7f9286191640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f928014be90 0x7f928014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:51252/0 (socket says 172.21.15.69:51252) 2024-03-25T19:48:53.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.348+0000 7f9286191640 1 -- 172.21.15.69:0/2012156074 learned_addr learned my addr 172.21.15.69:0/2012156074 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:53.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.361+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911839016 0 0) 0x7f92800b0a70 con 0x7f928014be90 2024-03-25T19:48:53.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.361+0000 7f928498e640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.361+0000 7f928498e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.361+0000 7f928498e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.361+0000 7f928498e640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:48:53.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.361+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c003520 con 0x7f928014be90 2024-03-25T19:48:53.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.372+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18897053 0 0) 0x7f92800b0d50 con 0x7f92800b1730 2024-03-25T19:48:53.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.372+0000 7f928498e640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.372+0000 7f928498e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.372+0000 7f928498e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.372+0000 7f928498e640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:48:53.372 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.372+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92800b0a70 con 0x7f92800b1730 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155756940 0 0) 0x7f926c003520 con 0x7f928014be90 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 >> v1:172.21.15.69:6790/0 conn(0x7f92800b1730 legacy=0x7f92800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 >> v1:172.21.15.69:6789/0 conn(0x7f92800b4d60 legacy=0x7f928013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 10 monclient: _finish_hunting 0 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 1 monclient: found mon.noname-c 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92800b1030 con 0x7f928014be90 2024-03-25T19:48:53.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 10 monclient: _finish_auth 0 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 10 monclient: _check_auth_tickets 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9278001a10 con 0x7f928014be90 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f9287bf4640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f9287bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f928498e640 10 monclient: dump: 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:53.375 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.376 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:53.376 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:53.376 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:53.376 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:53.376 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:53.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.373+0000 7f9287bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9278001b70 con 0x7f928014be90 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f928498e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f928498e640 1 -- 172.21.15.69:0/2012156074 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9278001f80 con 0x7f928014be90 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f9287bf4640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f9287bf4640 4 set_mon_vals no callback set 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f928498e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f928498e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f928498e640 10 monclient: dump: 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:53.382 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:53.383 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:53.383 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.383 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:53.383 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:53.383 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:53.383 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:53.383 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f9287bf4640 10 monclient: shutdown 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f9287bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f9287bf4640 1 -- 172.21.15.69:0/2012156074 >> v1:172.21.15.169:6789/0 conn(0x7f928014be90 legacy=0x7f928014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f9287bf4640 1 -- 172.21.15.69:0/2012156074 shutdown_connections 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.382+0000 7f9287bf4640 1 -- 172.21.15.69:0/2012156074 >> 172.21.15.69:0/2012156074 conn(0x7f92800617a0 msgr2=0x7f9280061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.383+0000 7f9287bf4640 1 -- 172.21.15.69:0/2012156074 shutdown_connections 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.383+0000 7f9287bf4640 1 -- 172.21.15.69:0/2012156074 wait complete. 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.383+0000 7f9287bf4640 10 monclient: build_initial_monmap 2024-03-25T19:48:53.385 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.383+0000 7f9287bf4640 10 monclient: monmap: 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 librados: starting msgr at 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 librados: starting objecter 2024-03-25T19:48:53.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 Processor -- start 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 -- start start 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 librados: setting wanted keys 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 librados: calling monclient init 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 10 monclient: init 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f92800b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f92800b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f928014be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92801381c0 con 0x7f928014be90 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9280137d70 con 0x7f92800b1730 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9280137f00 con 0x7f92800b4d60 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:53.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.384+0000 7f9287bf4640 10 monclient(hunting): authenticate will time out at 5345.653470s 2024-03-25T19:48:53.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.389+0000 7f9285990640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f92800b1730 0x7f9280140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:51316/0 (socket says 172.21.15.69:51316) 2024-03-25T19:48:53.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.389+0000 7f9285990640 1 -- 172.21.15.69:0/145666371 learned_addr learned my addr 172.21.15.69:0/145666371 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:48:53.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.391+0000 7f92767fc640 10 client.?.objecter ms_handle_connect 0x7f92800b1730 2024-03-25T19:48:53.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.391+0000 7f92767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.394+0000 7f92767fc640 10 client.?.objecter ms_handle_connect 0x7f92800b4d60 2024-03-25T19:48:53.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.394+0000 7f92767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.400+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589196776 0 0) 0x7f9280137f00 con 0x7f92800b4d60 2024-03-25T19:48:53.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.400+0000 7f92767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.400+0000 7f92767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.400+0000 7f92767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.400+0000 7f92767fc640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:48:53.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.400+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9264004eb0 con 0x7f92800b4d60 2024-03-25T19:48:53.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.403+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455598563 0 0) 0x7f9280137d70 con 0x7f92800b1730 2024-03-25T19:48:53.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.403+0000 7f92767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.403+0000 7f92767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.403+0000 7f92767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.403+0000 7f92767fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:48:53.404 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.403+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9280137f00 con 0x7f92800b1730 2024-03-25T19:48:53.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.407+0000 7f92767fc640 10 client.?.objecter ms_handle_connect 0x7f928014be90 2024-03-25T19:48:53.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.407+0000 7f92767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260128907 0 0) 0x7f9264004eb0 con 0x7f92800b4d60 2024-03-25T19:48:53.417 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9280137d70 con 0x7f92800b4d60 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9270001970 con 0x7f92800b4d60 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 10 monclient(hunting): dump: 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:53.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:53.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040095386 0 0) 0x7f92801381c0 con 0x7f928014be90 2024-03-25T19:48:53.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:48:53.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.417+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9264004eb0 con 0x7f928014be90 2024-03-25T19:48:53.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 162659370 0 0) 0x7f9280137d70 con 0x7f92800b4d60 2024-03-25T19:48:53.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 >> v1:172.21.15.169:6789/0 conn(0x7f92800b1730 legacy=0x7f9280140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 >> v1:172.21.15.69:6789/0 conn(0x7f928014be90 legacy=0x7f92801e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:53.426 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 1 monclient: found mon.c 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9280145040 con 0x7f92800b4d60 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f92767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 5 monclient: authenticate success, global_id 4577 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 10 monclient: _renew_subs 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9280145290 con 0x7f92800b4d60 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 10 monclient: _renew_subs 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92801369c0 con 0x7f92800b4d60 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.425+0000 7f9287bf4640 1 librados: init done 2024-03-25T19:48:53.427 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.426+0000 7f9287bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.426+0000 7f9287bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.426+0000 7f9287bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:53.428 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.426+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9280136c70 con 0x7f92800b4d60 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.721+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9270003610 con 0x7f92800b4d60 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.721+0000 7f92767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.721+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9270003a60 con 0x7f92800b4d60 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.721+0000 7f92767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.721+0000 7f92767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.721+0000 7f92767fc640 10 monclient: dump: 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:48:53.722 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.723 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:48:53.723 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:48:53.723 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:48:53.723 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:48:53.723 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:48:53.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.721+0000 7f9277fff640 4 set_mon_vals no callback set 2024-03-25T19:48:53.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.722+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f9270003c80 con 0x7f92800b4d60 2024-03-25T19:48:53.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.723+0000 7f9285990640 10 monclient: get_auth_request con 0x7f926406de00 auth_method 0 2024-03-25T19:48:53.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.723+0000 7f92767fc640 10 client.4577.objecter ms_handle_connect 0x7f926406de00 2024-03-25T19:48:53.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.727+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f9270053630 con 0x7f92800b4d60 2024-03-25T19:48:53.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.727+0000 7f92767fc640 10 client.4577.objecter ms_dispatch 0x7f9280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:53.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.727+0000 7f92767fc640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:53.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.727+0000 7f92767fc640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:53.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.727+0000 7f92767fc640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-25T19:48:53.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.727+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f927004f140 con 0x7f92800b4d60 2024-03-25T19:48:53.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.727+0000 7f92767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.727+0000 7f92767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.949+0000 7f9287bf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.949+0000 7f9287bf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.949+0000 7f9287bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.949+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9280138370 con 0x7f92800b4d60 2024-03-25T19:48:53.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.973+0000 7f92767fc640 1 -- 172.21.15.69:0/145666371 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f9280138370 con 0x7f92800b4d60 2024-03-25T19:48:53.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.973+0000 7f92767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.974 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.973+0000 7f92767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:53.974 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:48:53.974 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 10 librados: watch_flush enter 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 10 librados: watch_flush exit 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 10 client.4577.objecter successfully canceled tick 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 >> v1:172.21.15.169:6800/3072548259 conn(0x7f926406de00 legacy=0x7f92640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 10 monclient: shutdown 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 >> v1:172.21.15.69:6790/0 conn(0x7f92800b4d60 legacy=0x7f92801360d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 shutdown_connections 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 >> 172.21.15.69:0/145666371 conn(0x7f92800617a0 msgr2=0x7f92800b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 shutdown_connections 2024-03-25T19:48:53.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 1 -- 172.21.15.69:0/145666371 wait complete. 2024-03-25T19:48:53.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:48:53.976+0000 7f9287bf4640 1 librados: shutdown 2024-03-25T19:48:53.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:00.795 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:00.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.898+0000 7fb5f489a640 10 monclient: get_monmap_and_config 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.898+0000 7fb5f489a640 10 monclient: build_initial_monmap 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.898+0000 7fb5f489a640 10 monclient: monmap: 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:49:00.899644+0000 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:49:00.899644+0000 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:49:00.899 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:00.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.899+0000 7fb5f489a640 1 Processor -- start 2024-03-25T19:49:00.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.899+0000 7fb5f489a640 1 -- start start 2024-03-25T19:49:00.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.899+0000 7fb5f489a640 10 monclient: init 2024-03-25T19:49:00.900 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 10 monclient(hunting): picked mon.noname-c con 0x7fb5f00a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 10 monclient(hunting): picked mon.noname-b con 0x7fb5f013f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 10 monclient(hunting): picked mon.noname-a con 0x7fb5f01430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f0142520 con 0x7fb5f01430c0 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f00b28a0 con 0x7fb5f00a98f0 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f00b2a30 con 0x7fb5f013f710 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:00.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.900+0000 7fb5f489a640 10 monclient(hunting): authenticate will time out at 5353.168832s 2024-03-25T19:49:00.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.903+0000 7fb5ee59c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb5f00a98f0 0x7fb5f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:34996/0 (socket says 172.21.15.69:34996) 2024-03-25T19:49:00.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.903+0000 7fb5ee59c640 1 -- 172.21.15.69:0/3525421399 learned_addr learned my addr 172.21.15.69:0/3525421399 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.928+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181746030 0 0) 0x7fb5f0142520 con 0x7fb5f01430c0 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.928+0000 7fb5ed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.928+0000 7fb5ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.928+0000 7fb5ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.928+0000 7fb5ed59a640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:00.928+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8003520 con 0x7fb5f01430c0 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127828407 0 0) 0x7fb5d8003520 con 0x7fb5f01430c0 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 >> v1:172.21.15.69:6790/0 conn(0x7fb5f013f710 legacy=0x7fb5f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 >> v1:172.21.15.169:6789/0 conn(0x7fb5f00a98f0 legacy=0x7fb5f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 10 monclient: _finish_hunting 0 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 1 monclient: found mon.noname-a 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f00b44c0 con 0x7fb5f01430c0 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 10 monclient: _finish_auth 0 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 10 monclient: _check_auth_tickets 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb5e8001a10 con 0x7fb5f01430c0 2024-03-25T19:49:01.003 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5f489a640 5 monclient: authenticate success, global_id 4602 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5f489a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5ed59a640 10 monclient: dump: 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.002+0000 7fb5f489a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.017+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5e8001b70 con 0x7fb5f01430c0 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.017+0000 7fb5ed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.017+0000 7fb5ed59a640 1 -- 172.21.15.69:0/3525421399 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb5e8001f80 con 0x7fb5f01430c0 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.017+0000 7fb5f489a640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.017+0000 7fb5f489a640 4 set_mon_vals no callback set 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.017+0000 7fb5ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.017+0000 7fb5ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.017+0000 7fb5ed59a640 10 monclient: dump: 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:01.018 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.018+0000 7fb5f489a640 10 monclient: shutdown 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.018+0000 7fb5f489a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.018+0000 7fb5f489a640 1 -- 172.21.15.69:0/3525421399 >> v1:172.21.15.69:6789/0 conn(0x7fb5f01430c0 legacy=0x7fb5f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.018+0000 7fb5f489a640 1 -- 172.21.15.69:0/3525421399 shutdown_connections 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.018+0000 7fb5f489a640 1 -- 172.21.15.69:0/3525421399 >> 172.21.15.69:0/3525421399 conn(0x7fb5f00b7ea0 msgr2=0x7fb5f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.018+0000 7fb5f489a640 1 -- 172.21.15.69:0/3525421399 shutdown_connections 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.018+0000 7fb5f489a640 1 -- 172.21.15.69:0/3525421399 wait complete. 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 10 monclient: build_initial_monmap 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 10 monclient: monmap: 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:49:01.020 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 1 librados: starting msgr at 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 1 librados: starting objecter 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 1 Processor -- start 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 1 -- start start 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 1 librados: setting wanted keys 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 1 librados: calling monclient init 2024-03-25T19:49:01.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.019+0000 7fb5f489a640 10 monclient: init 2024-03-25T19:49:01.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:01.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:01.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 10 monclient(hunting): picked mon.noname-a con 0x7fb5f00a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:01.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 10 monclient(hunting): picked mon.noname-b con 0x7fb5f013f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:01.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 10 monclient(hunting): picked mon.noname-c con 0x7fb5f01430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:01.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f0138b60 con 0x7fb5f00a98f0 2024-03-25T19:49:01.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f0137e40 con 0x7fb5f013f710 2024-03-25T19:49:01.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f01d82f0 con 0x7fb5f01430c0 2024-03-25T19:49:01.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:01.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.020+0000 7fb5f489a640 10 monclient(hunting): authenticate will time out at 5353.289529s 2024-03-25T19:49:01.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.034+0000 7fb5eed9d640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fb5f01430c0 0x7fb5f0133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:40258/0 (socket says 172.21.15.69:40258) 2024-03-25T19:49:01.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.034+0000 7fb5eed9d640 1 -- 172.21.15.69:0/3066658333 learned_addr learned my addr 172.21.15.69:0/3066658333 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:01.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.034+0000 7fb5deffd640 10 client.?.objecter ms_handle_connect 0x7fb5f01430c0 2024-03-25T19:49:01.034 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.034+0000 7fb5deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:01.041 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.041+0000 7fb5deffd640 10 client.?.objecter ms_handle_connect 0x7fb5f00a98f0 2024-03-25T19:49:01.041 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.041+0000 7fb5deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:01.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.042+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567075554 0 0) 0x7fb5f01d82f0 con 0x7fb5f01430c0 2024-03-25T19:49:01.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.042+0000 7fb5deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.042+0000 7fb5deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.042+0000 7fb5deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.042+0000 7fb5deffd640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:49:01.042 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.042+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d0004eb0 con 0x7fb5f01430c0 2024-03-25T19:49:01.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.046+0000 7fb5deffd640 10 client.?.objecter ms_handle_connect 0x7fb5f013f710 2024-03-25T19:49:01.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.046+0000 7fb5deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.054+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3905846942 0 0) 0x7fb5d0004eb0 con 0x7fb5f01430c0 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.054+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5f01d82f0 con 0x7fb5f01430c0 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.054+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb5e8001f40 con 0x7fb5f01430c0 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.054+0000 7fb5deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.054+0000 7fb5deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.054+0000 7fb5deffd640 10 monclient(hunting): dump: 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:01.056 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:01.056 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:01.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.054+0000 7fb5deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.070+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052280384 0 0) 0x7fb5f0137e40 con 0x7fb5f013f710 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.070+0000 7fb5deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.070+0000 7fb5deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.070+0000 7fb5deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.070+0000 7fb5deffd640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.070+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d0004eb0 con 0x7fb5f013f710 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.072+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460473969 0 0) 0x7fb5f0138b60 con 0x7fb5f00a98f0 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.072+0000 7fb5deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.072+0000 7fb5deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.072+0000 7fb5deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.072+0000 7fb5deffd640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.072+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f0137e40 con 0x7fb5f00a98f0 2024-03-25T19:49:01.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2866431954 0 0) 0x7fb5f01d82f0 con 0x7fb5f01430c0 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 >> v1:172.21.15.169:6789/0 conn(0x7fb5f013f710 legacy=0x7fb5f01d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 >> v1:172.21.15.69:6789/0 conn(0x7fb5f00a98f0 legacy=0x7fb5f0137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 1 monclient: found mon.c 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0134e70 con 0x7fb5f01430c0 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 5 monclient: authenticate success, global_id 4583 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 10 monclient: _renew_subs 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:01.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f01350c0 con 0x7fb5f01430c0 2024-03-25T19:49:01.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:01.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 10 monclient: _renew_subs 2024-03-25T19:49:01.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:01.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f0134800 con 0x7fb5f01430c0 2024-03-25T19:49:01.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.075+0000 7fb5f489a640 1 librados: init done 2024-03-25T19:49:01.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.076+0000 7fb5f489a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.076+0000 7fb5f489a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.078 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.076+0000 7fb5f489a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:01.078 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.076+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f0134ab0 con 0x7fb5f01430c0 2024-03-25T19:49:01.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.381+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5e8003c20 con 0x7fb5f01430c0 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.381+0000 7fb5deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.381+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb5e8003ff0 con 0x7fb5f01430c0 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.381+0000 7fb5deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.381+0000 7fb5deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.381+0000 7fb5deffd640 10 monclient: dump: 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:01.382 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:01.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.381+0000 7fb5ecd99640 4 set_mon_vals no callback set 2024-03-25T19:49:01.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.382+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fb5e8004210 con 0x7fb5f01430c0 2024-03-25T19:49:01.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.383+0000 7fb5ee59c640 10 monclient: get_auth_request con 0x7fb5d006de80 auth_method 0 2024-03-25T19:49:01.383 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.383+0000 7fb5deffd640 10 client.4583.objecter ms_handle_connect 0x7fb5d006de80 2024-03-25T19:49:01.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.387+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fb5e8054bd0 con 0x7fb5f01430c0 2024-03-25T19:49:01.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.387+0000 7fb5deffd640 10 client.4583.objecter ms_dispatch 0x7fb5f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:01.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.387+0000 7fb5deffd640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:01.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.387+0000 7fb5deffd640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:01.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.387+0000 7fb5deffd640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-25T19:49:01.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.387+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5e80506e0 con 0x7fb5f01430c0 2024-03-25T19:49:01.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.387+0000 7fb5deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.387+0000 7fb5deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:01.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.638+0000 7fb5f489a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.638+0000 7fb5f489a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.638+0000 7fb5f489a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:01.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.638+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5f00abb30 con 0x7fb5f01430c0 2024-03-25T19:49:01.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.663+0000 7fb5deffd640 1 -- 172.21.15.69:0/3066658333 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fb5e8052610 con 0x7fb5f01430c0 2024-03-25T19:49:01.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.663+0000 7fb5deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.663+0000 7fb5deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:01.664 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:49:01.664 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:01.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 10 librados: watch_flush enter 2024-03-25T19:49:01.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 10 librados: watch_flush exit 2024-03-25T19:49:01.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-25T19:49:01.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 10 client.4583.objecter successfully canceled tick 2024-03-25T19:49:01.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 >> v1:172.21.15.169:6800/3072548259 conn(0x7fb5d006de80 legacy=0x7fb5d0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 10 monclient: shutdown 2024-03-25T19:49:01.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 >> v1:172.21.15.69:6790/0 conn(0x7fb5f01430c0 legacy=0x7fb5f0133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 shutdown_connections 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.665+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 >> 172.21.15.69:0/3066658333 conn(0x7fb5f00b7ea0 msgr2=0x7fb5f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.666+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 shutdown_connections 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.666+0000 7fb5f489a640 1 -- 172.21.15.69:0/3066658333 wait complete. 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:01.666+0000 7fb5f489a640 1 librados: shutdown 2024-03-25T19:49:01.678 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:08.426 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:46.390 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.478 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:46.442 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.485 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:08.515 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:46.479 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.529 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:46.483 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.531 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:46.495 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.534 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:46.488 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.540 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:46.504 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.557 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:46.511 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.569 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:46.533 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:08.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.588+0000 7f32fcb54640 10 monclient: get_monmap_and_config 2024-03-25T19:49:08.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.588+0000 7f32fcb54640 10 monclient: build_initial_monmap 2024-03-25T19:49:08.588 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.588+0000 7f32fcb54640 10 monclient: monmap: 2024-03-25T19:49:08.588 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:08.588 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:08.589 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:49:08.589656+0000 2024-03-25T19:49:08.589 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:49:08.589656+0000 2024-03-25T19:49:08.589 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:08.589 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:08.589 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:08.589 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:49:08.589 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:49:08.589 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:08.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 1 Processor -- start 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 1 -- start start 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 10 monclient: init 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 10 monclient(hunting): picked mon.noname-c con 0x7f32f80b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 10 monclient(hunting): picked mon.noname-a con 0x7f32f80b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 10 monclient(hunting): picked mon.noname-b con 0x7f32f814be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.589+0000 7f32fcb54640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80af940 con 0x7f32f80b1730 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.590+0000 7f32fcb54640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80b0a70 con 0x7f32f80b4d60 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.590+0000 7f32fcb54640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80b0d50 con 0x7f32f814be90 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.590+0000 7f32fcb54640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:08.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.590+0000 7f32fcb54640 10 monclient(hunting): authenticate will time out at 5360.858671s 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.617+0000 7f32f5d9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f32f80b1730 0x7f32f80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50550/0 (socket says 172.21.15.69:50550) 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.617+0000 7f32f5d9b640 1 -- 172.21.15.69:0/1828829190 learned_addr learned my addr 172.21.15.69:0/1828829190 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:08.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.656+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387265125 0 0) 0x7f32f80af940 con 0x7f32f80b1730 2024-03-25T19:49:08.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.656+0000 7f32f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.656+0000 7f32f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.656+0000 7f32f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.656+0000 7f32f559a640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:49:08.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.656+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e0003520 con 0x7f32f80b1730 2024-03-25T19:49:08.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.663+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122890871 0 0) 0x7f32f80b0a70 con 0x7f32f80b4d60 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.664+0000 7f32f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.664+0000 7f32f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.664+0000 7f32f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.664+0000 7f32f559a640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.664+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f80af940 con 0x7f32f80b4d60 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567979627 0 0) 0x7f32e0003520 con 0x7f32f80b1730 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 >> v1:172.21.15.69:6790/0 conn(0x7f32f814be90 legacy=0x7f32f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 >> v1:172.21.15.169:6789/0 conn(0x7f32f80b4d60 legacy=0x7f32f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 10 monclient: _finish_hunting 0 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 1 monclient: found mon.noname-a 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80b1030 con 0x7f32f80b1730 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 10 monclient: _finish_auth 0 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 10 monclient: _check_auth_tickets 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f32ec001690 con 0x7f32f80b1730 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32fcb54640 5 monclient: authenticate success, global_id 4608 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32fcb54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32f559a640 10 monclient: dump: 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:08.686 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:08.687 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:08.687 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:08.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.684+0000 7f32fcb54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:08.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.719+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32ec0017f0 con 0x7f32f80b1730 2024-03-25T19:49:08.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.719+0000 7f32f559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.719+0000 7f32f559a640 1 -- 172.21.15.69:0/1828829190 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f32ec001c00 con 0x7f32f80b1730 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.719+0000 7f32fcb54640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.719+0000 7f32fcb54640 4 set_mon_vals no callback set 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.719+0000 7f32f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.719+0000 7f32f559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.719+0000 7f32f559a640 10 monclient: dump: 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:08.720 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:08.721 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:08.721 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:08.721 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.720+0000 7f32fcb54640 10 monclient: shutdown 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.720+0000 7f32fcb54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.720+0000 7f32fcb54640 1 -- 172.21.15.69:0/1828829190 >> v1:172.21.15.69:6789/0 conn(0x7f32f80b1730 legacy=0x7f32f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.720+0000 7f32fcb54640 1 -- 172.21.15.69:0/1828829190 shutdown_connections 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.720+0000 7f32fcb54640 1 -- 172.21.15.69:0/1828829190 >> 172.21.15.69:0/1828829190 conn(0x7f32f80617a0 msgr2=0x7f32f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.720+0000 7f32fcb54640 1 -- 172.21.15.69:0/1828829190 shutdown_connections 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.720+0000 7f32fcb54640 1 -- 172.21.15.69:0/1828829190 wait complete. 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.721+0000 7f32fcb54640 10 monclient: build_initial_monmap 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.721+0000 7f32fcb54640 10 monclient: monmap: 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.721+0000 7f32fcb54640 1 librados: starting msgr at 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.721+0000 7f32fcb54640 1 librados: starting objecter 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 1 Processor -- start 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 1 -- start start 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 1 librados: setting wanted keys 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 1 librados: calling monclient init 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 10 monclient: init 2024-03-25T19:49:08.724 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 10 monclient(hunting): picked mon.noname-c con 0x7f32f80b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 10 monclient(hunting): picked mon.noname-a con 0x7f32f80b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 10 monclient(hunting): picked mon.noname-b con 0x7f32f814be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f8144ea0 con 0x7f32f80b4d60 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f8136f90 con 0x7f32f814be90 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f8144a30 con 0x7f32f80b1730 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:08.725 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.722+0000 7f32fcb54640 10 monclient(hunting): authenticate will time out at 5360.991548s 2024-03-25T19:49:08.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.726+0000 7f32f6d9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f32f814be90 0x7f32f812e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:39900/0 (socket says 172.21.15.69:39900) 2024-03-25T19:49:08.726 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.726+0000 7f32f6d9d640 1 -- 172.21.15.69:0/3947979033 learned_addr learned my addr 172.21.15.69:0/3947979033 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:08.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.728+0000 7f32e6ffd640 10 client.?.objecter ms_handle_connect 0x7f32f814be90 2024-03-25T19:49:08.728 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.728+0000 7f32e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.731+0000 7f32e6ffd640 10 client.?.objecter ms_handle_connect 0x7f32f80b1730 2024-03-25T19:49:08.731 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.731+0000 7f32e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.788 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.787+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200535950 0 0) 0x7f32f8136f90 con 0x7f32f814be90 2024-03-25T19:49:08.788 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.787+0000 7f32e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.788 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.787+0000 7f32e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.788 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.787+0000 7f32e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.788 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.787+0000 7f32e6ffd640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:49:08.788 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.787+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d8004eb0 con 0x7f32f814be90 2024-03-25T19:49:08.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.789+0000 7f32e6ffd640 10 client.?.objecter ms_handle_connect 0x7f32f80b4d60 2024-03-25T19:49:08.789 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.789+0000 7f32e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.798+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2575162630 0 0) 0x7f32d8004eb0 con 0x7f32f814be90 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.798+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32f8136f90 con 0x7f32f814be90 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.798+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f32f0001c30 con 0x7f32f814be90 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.798+0000 7f32e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.798+0000 7f32e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.798+0000 7f32e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:08.798 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:08.799 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:08.799 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.799 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:08.799 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:08.799 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:08.799 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:08.799 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:08.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.798+0000 7f32e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:08.810 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3079217268 0 0) 0x7f32f8136f90 con 0x7f32f814be90 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 >> v1:172.21.15.69:6790/0 conn(0x7f32f80b1730 legacy=0x7f32f813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 >> v1:172.21.15.69:6789/0 conn(0x7f32f80b4d60 legacy=0x7f32f8137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 1 monclient: found mon.b 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f8144bc0 con 0x7f32f814be90 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 10 monclient: _renew_subs 2024-03-25T19:49:08.811 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:08.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f812e7a0 con 0x7f32f814be90 2024-03-25T19:49:08.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:08.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 10 monclient: _renew_subs 2024-03-25T19:49:08.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:08.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f81456f0 con 0x7f32f814be90 2024-03-25T19:49:08.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.810+0000 7f32fcb54640 1 librados: init done 2024-03-25T19:49:08.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.811+0000 7f32fcb54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:08.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.811+0000 7f32fcb54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:08.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.811+0000 7f32fcb54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:08.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:08.811+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f80627b0 con 0x7f32f814be90 2024-03-25T19:49:09.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.089+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32f0003970 con 0x7f32f814be90 2024-03-25T19:49:09.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.089+0000 7f32e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:09.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.089+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f32f0003dc0 con 0x7f32f814be90 2024-03-25T19:49:09.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.089+0000 7f32e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:09.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.089+0000 7f32e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:09.090 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.089+0000 7f32e6ffd640 10 monclient: dump: 2024-03-25T19:49:09.090 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:09.090 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:09.091 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.089+0000 7f32f4d99640 4 set_mon_vals no callback set 2024-03-25T19:49:09.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.090+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f32f0003fe0 con 0x7f32f814be90 2024-03-25T19:49:09.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.091+0000 7f32f659c640 10 monclient: get_auth_request con 0x7f32d806dd90 auth_method 0 2024-03-25T19:49:09.092 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.091+0000 7f32e6ffd640 10 client.4579.objecter ms_handle_connect 0x7f32d806dd90 2024-03-25T19:49:09.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.108+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f32f00549d0 con 0x7f32f814be90 2024-03-25T19:49:09.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.108+0000 7f32e6ffd640 10 client.4579.objecter ms_dispatch 0x7f32f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:09.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.108+0000 7f32e6ffd640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:09.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.108+0000 7f32e6ffd640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:09.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.108+0000 7f32e6ffd640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T19:49:09.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.109+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32f00504e0 con 0x7f32f814be90 2024-03-25T19:49:09.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.109+0000 7f32e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:09.109 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.109+0000 7f32e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:09.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.362+0000 7f32fcb54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.362+0000 7f32fcb54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.362+0000 7f32fcb54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:09.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.362+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32f812e930 con 0x7f32f814be90 2024-03-25T19:49:09.398 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.398+0000 7f32e6ffd640 1 -- 172.21.15.69:0/3947979033 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f32f0052410 con 0x7f32f814be90 2024-03-25T19:49:09.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.398+0000 7f32e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.398+0000 7f32e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:09.399 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:49:09.399 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:09.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.401+0000 7f32fcb54640 10 librados: watch_flush enter 2024-03-25T19:49:09.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.401+0000 7f32fcb54640 10 librados: watch_flush exit 2024-03-25T19:49:09.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.401+0000 7f32fcb54640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T19:49:09.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.401+0000 7f32fcb54640 10 client.4579.objecter successfully canceled tick 2024-03-25T19:49:09.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.402+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 >> v1:172.21.15.169:6800/3072548259 conn(0x7f32d806dd90 legacy=0x7f32d8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:09.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.402+0000 7f32fcb54640 10 monclient: shutdown 2024-03-25T19:49:09.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.402+0000 7f32fcb54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:09.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.402+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 >> v1:172.21.15.169:6789/0 conn(0x7f32f814be90 legacy=0x7f32f812e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:09.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.402+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 shutdown_connections 2024-03-25T19:49:09.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.402+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 >> 172.21.15.69:0/3947979033 conn(0x7f32f80617a0 msgr2=0x7f32f813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:09.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.403+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 shutdown_connections 2024-03-25T19:49:09.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.403+0000 7f32fcb54640 1 -- 172.21.15.69:0/3947979033 wait complete. 2024-03-25T19:49:09.403 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:09.403+0000 7f32fcb54640 1 librados: shutdown 2024-03-25T19:49:09.416 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:16.223 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:16.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.338+0000 7f484e1f4640 10 monclient: get_monmap_and_config 2024-03-25T19:49:16.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.338+0000 7f484e1f4640 10 monclient: build_initial_monmap 2024-03-25T19:49:16.337 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.338+0000 7f484e1f4640 10 monclient: monmap: 2024-03-25T19:49:16.337 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:49:16.338387+0000 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:49:16.338387+0000 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:49:16.338 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:16.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 1 Processor -- start 2024-03-25T19:49:16.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 1 -- start start 2024-03-25T19:49:16.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 10 monclient: init 2024-03-25T19:49:16.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:16.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:16.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 10 monclient(hunting): picked mon.noname-b con 0x7f48480b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:16.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 10 monclient(hunting): picked mon.noname-a con 0x7f48480b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:16.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 10 monclient(hunting): picked mon.noname-c con 0x7f484814be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:16.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.339+0000 7f484e1f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480af940 con 0x7f48480b1730 2024-03-25T19:49:16.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.340+0000 7f484e1f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480b0a70 con 0x7f484814be90 2024-03-25T19:49:16.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.340+0000 7f484e1f4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480b0d50 con 0x7f48480b4d60 2024-03-25T19:49:16.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.340+0000 7f484e1f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:16.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.340+0000 7f484e1f4640 10 monclient(hunting): authenticate will time out at 5368.607629s 2024-03-25T19:49:16.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.349+0000 7f48477fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f48480b4d60 0x7f484813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:47756/0 (socket says 172.21.15.69:47756) 2024-03-25T19:49:16.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.349+0000 7f48477fe640 1 -- 172.21.15.69:0/4077671014 learned_addr learned my addr 172.21.15.69:0/4077671014 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:16.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.363+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571164978 0 0) 0x7f48480b0d50 con 0x7f48480b4d60 2024-03-25T19:49:16.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.363+0000 7f48467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.363+0000 7f48467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.363+0000 7f48467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.363+0000 7f48467fc640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:49:16.363 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.363+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4830003520 con 0x7f48480b4d60 2024-03-25T19:49:16.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.375+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722765418 0 0) 0x7f48480b0a70 con 0x7f484814be90 2024-03-25T19:49:16.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.375+0000 7f48467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.375+0000 7f48467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.375+0000 7f48467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.375+0000 7f48467fc640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:49:16.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.375+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48480b0d50 con 0x7f484814be90 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3310287193 0 0) 0x7f4830003520 con 0x7f48480b4d60 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 >> v1:172.21.15.169:6789/0 conn(0x7f484814be90 legacy=0x7f484814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 >> v1:172.21.15.69:6789/0 conn(0x7f48480b1730 legacy=0x7f48480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 1 monclient: found mon.noname-b 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48480b1030 con 0x7f48480b4d60 2024-03-25T19:49:16.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:16.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4838001690 con 0x7f48480b4d60 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.376+0000 7f48467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.377+0000 7f48467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.377+0000 7f48467fc640 10 monclient: dump: 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.378 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:16.379 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:16.379 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:16.379 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:16.379 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:16.379 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.377+0000 7f484e1f4640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:49:16.379 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.377+0000 7f484e1f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.386+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48380017f0 con 0x7f48480b4d60 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.386+0000 7f48467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.386+0000 7f48467fc640 1 -- 172.21.15.69:0/4077671014 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4838001c00 con 0x7f48480b4d60 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.386+0000 7f48467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.386+0000 7f48467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.386+0000 7f48467fc640 10 monclient: dump: 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:16.386 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:16.387 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:16.387 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:16.387 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:16.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.387+0000 7f484e1f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:16.387 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.387+0000 7f484e1f4640 4 set_mon_vals no callback set 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.387+0000 7f484e1f4640 10 monclient: shutdown 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.387+0000 7f484e1f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.387+0000 7f484e1f4640 1 -- 172.21.15.69:0/4077671014 >> v1:172.21.15.69:6790/0 conn(0x7f48480b4d60 legacy=0x7f484813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.388+0000 7f484e1f4640 1 -- 172.21.15.69:0/4077671014 shutdown_connections 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.388+0000 7f484e1f4640 1 -- 172.21.15.69:0/4077671014 >> 172.21.15.69:0/4077671014 conn(0x7f48480617a0 msgr2=0x7f4848061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.388+0000 7f484e1f4640 1 -- 172.21.15.69:0/4077671014 shutdown_connections 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.388+0000 7f484e1f4640 1 -- 172.21.15.69:0/4077671014 wait complete. 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 10 monclient: build_initial_monmap 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 10 monclient: monmap: 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:16.388 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:16.389 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:49:16.389 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:49:16.389 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:16.389 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:49:16.389 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:16.389 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:49:16.389 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:49:16.389 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:16.390 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 1 librados: starting msgr at 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 1 librados: starting objecter 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 1 Processor -- start 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 1 -- start start 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 1 librados: setting wanted keys 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 1 librados: calling monclient init 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.389+0000 7f484e1f4640 10 monclient: init 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 10 monclient(hunting): picked mon.noname-a con 0x7f48480b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 10 monclient(hunting): picked mon.noname-c con 0x7f48480b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 10 monclient(hunting): picked mon.noname-b con 0x7f484814be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:16.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484813e050 con 0x7f48480b1730 2024-03-25T19:49:16.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48481452a0 con 0x7f484814be90 2024-03-25T19:49:16.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484813db70 con 0x7f48480b4d60 2024-03-25T19:49:16.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:16.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.390+0000 7f484e1f4640 10 monclient(hunting): authenticate will time out at 5368.658305s 2024-03-25T19:49:16.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.395+0000 7f48477fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f48480b1730 0x7f48481d78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60388/0 (socket says 172.21.15.69:60388) 2024-03-25T19:49:16.394 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.395+0000 7f48477fe640 1 -- 172.21.15.69:0/457307169 learned_addr learned my addr 172.21.15.69:0/457307169 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:16.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.401+0000 7f482ffff640 10 client.?.objecter ms_handle_connect 0x7f48480b1730 2024-03-25T19:49:16.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.401+0000 7f482ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.403+0000 7f482ffff640 10 client.?.objecter ms_handle_connect 0x7f484814be90 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.403+0000 7f482ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.405 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.406+0000 7f482ffff640 10 client.?.objecter ms_handle_connect 0x7f48480b4d60 2024-03-25T19:49:16.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.407+0000 7f482ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.410+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983158640 0 0) 0x7f484813e050 con 0x7f48480b1730 2024-03-25T19:49:16.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.410+0000 7f482ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.410+0000 7f482ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.410+0000 7f482ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.410+0000 7f482ffff640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:49:16.410 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.410+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4834004eb0 con 0x7f48480b1730 2024-03-25T19:49:16.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.416+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074375346 0 0) 0x7f48481452a0 con 0x7f484814be90 2024-03-25T19:49:16.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.416+0000 7f482ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.416+0000 7f482ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.416+0000 7f482ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.416+0000 7f482ffff640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:49:16.415 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.416+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f484813e050 con 0x7f484814be90 2024-03-25T19:49:16.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.419+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716317794 0 0) 0x7f484813db70 con 0x7f48480b4d60 2024-03-25T19:49:16.418 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.419+0000 7f482ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.419+0000 7f482ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.419+0000 7f482ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.419+0000 7f482ffff640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:49:16.419 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.419+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48481452a0 con 0x7f48480b4d60 2024-03-25T19:49:16.434 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.435+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1388602565 0 0) 0x7f4834004eb0 con 0x7f48480b1730 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.435+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f484813db70 con 0x7f48480b1730 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.435+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4838001bc0 con 0x7f48480b1730 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.435+0000 7f482ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.435+0000 7f482ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.435+0000 7f482ffff640 10 monclient(hunting): dump: 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:16.436 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:16.436 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:16.436 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.435+0000 7f482ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.436+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1955208021 0 0) 0x7f48481452a0 con 0x7f48480b4d60 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.436+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4834004eb0 con 0x7f48480b4d60 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.436+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f483c001830 con 0x7f48480b4d60 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.436+0000 7f482ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.436+0000 7f482ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.436+0000 7f482ffff640 10 monclient(hunting): dump: 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.437 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.438 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.438 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:16.438 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:16.438 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:16.438 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:16.438 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.448+0000 7f48477fe640 1 -- 172.21.15.69:0/457307169 >> v1:172.21.15.69:6789/0 conn(0x7f48480b1730 legacy=0x7f48481d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.448+0000 7f48477fe640 1 -- 172.21.15.69:0/457307169 >> v1:172.21.15.69:6789/0 conn(0x7f48480b1730 legacy=0x7f48481d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.448+0000 7f48477fe640 1 --1- 172.21.15.69:0/457307169 >> v1:172.21.15.69:6789/0 conn(0x7f48480b1730 0x7f48481d78a0 :-1 s=OPENED pgs=64255 cs=1 l=1).handle_message read tag failed 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.448+0000 7f48477fe640 1 --1- 172.21.15.69:0/457307169 >> v1:172.21.15.69:6789/0 conn(0x7f48480b1730 0x7f48481d78a0 :-1 s=OPENED pgs=64255 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:49:16.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.448+0000 7f482ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.69:6789/0 2024-03-25T19:49:16.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.571+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 263713589 0 0) 0x7f484813e050 con 0x7f484814be90 2024-03-25T19:49:16.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.571+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48481452a0 con 0x7f484814be90 2024-03-25T19:49:16.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.571+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4840001b90 con 0x7f484814be90 2024-03-25T19:49:16.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.571+0000 7f482ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.571+0000 7f482ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.571+0000 7f482ffff640 10 monclient(hunting): dump: 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:16.571 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2604767852 0 0) 0x7f4834004eb0 con 0x7f48480b4d60 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 >> v1:172.21.15.169:6789/0 conn(0x7f484814be90 legacy=0x7f48481e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 >> v1:172.21.15.69:6789/0 conn(0x7f48480b1730 legacy=0x7f48481d78a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 1 monclient: found mon.c 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484813dd00 con 0x7f48480b4d60 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 10 monclient: _finish_auth 0 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f482ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 10 monclient: _renew_subs 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48481455a0 con 0x7f48480b4d60 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 10 monclient: _renew_subs 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4848135e80 con 0x7f48480b4d60 2024-03-25T19:49:16.575 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.574+0000 7f484e1f4640 1 librados: init done 2024-03-25T19:49:16.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.575+0000 7f484e1f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.575+0000 7f484e1f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.575+0000 7f484e1f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:16.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.575+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48480627b0 con 0x7f48480b4d60 2024-03-25T19:49:16.872 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.873+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f483c001d40 con 0x7f48480b4d60 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.873+0000 7f482ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.873+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f483c003790 con 0x7f48480b4d60 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.873+0000 7f482ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.873+0000 7f482ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.873+0000 7f482ffff640 10 monclient: dump: 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.873 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:16.874 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:16.874 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:16.874 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:16.874 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:16.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.873+0000 7f4845ffb640 4 set_mon_vals no callback set 2024-03-25T19:49:16.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.874+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f483c0039b0 con 0x7f48480b4d60 2024-03-25T19:49:16.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.875+0000 7f48477fe640 10 monclient: get_auth_request con 0x7f483406e0b0 auth_method 0 2024-03-25T19:49:16.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.875+0000 7f482ffff640 10 client.4589.objecter ms_handle_connect 0x7f483406e0b0 2024-03-25T19:49:16.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.881+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f483c053590 con 0x7f48480b4d60 2024-03-25T19:49:16.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.881+0000 7f482ffff640 10 client.4589.objecter ms_dispatch 0x7f4848000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:16.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.881+0000 7f482ffff640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:16.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.881+0000 7f482ffff640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:16.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.881+0000 7f482ffff640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T19:49:16.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.881+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f483c04eed0 con 0x7f48480b4d60 2024-03-25T19:49:16.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.881+0000 7f482ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:16.881+0000 7f482ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:17.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.142+0000 7f484e1f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.142+0000 7f484e1f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.142+0000 7f484e1f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:17.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.142+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4848069f40 con 0x7f48480b4d60 2024-03-25T19:49:17.173 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.174+0000 7f482ffff640 1 -- 172.21.15.69:0/457307169 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f483c050fd0 con 0x7f48480b4d60 2024-03-25T19:49:17.173 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.174+0000 7f482ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.174+0000 7f482ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:17.174 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:49:17.174 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:17.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.176+0000 7f484e1f4640 10 librados: watch_flush enter 2024-03-25T19:49:17.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.176+0000 7f484e1f4640 10 librados: watch_flush exit 2024-03-25T19:49:17.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.176+0000 7f484e1f4640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T19:49:17.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.176+0000 7f484e1f4640 10 client.4589.objecter successfully canceled tick 2024-03-25T19:49:17.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.176+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 >> v1:172.21.15.169:6800/3072548259 conn(0x7f483406e0b0 legacy=0x7f4834070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:17.176 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.176+0000 7f484e1f4640 10 monclient: shutdown 2024-03-25T19:49:17.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.176+0000 7f484e1f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:17.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.176+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 >> v1:172.21.15.69:6790/0 conn(0x7f48480b4d60 legacy=0x7f48481d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:17.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.177+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 shutdown_connections 2024-03-25T19:49:17.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.177+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 >> 172.21.15.69:0/457307169 conn(0x7f48480617a0 msgr2=0x7f484813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:17.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.177+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 shutdown_connections 2024-03-25T19:49:17.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.177+0000 7f484e1f4640 1 -- 172.21.15.69:0/457307169 wait complete. 2024-03-25T19:49:17.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:17.177+0000 7f484e1f4640 1 librados: shutdown 2024-03-25T19:49:17.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:23.992 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:24.105 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.105+0000 7fb1fd2c3640 10 monclient: get_monmap_and_config 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.105+0000 7fb1fd2c3640 10 monclient: build_initial_monmap 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.105+0000 7fb1fd2c3640 10 monclient: monmap: 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:49:24.107039+0000 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:49:24.107039+0000 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:49:24.106 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:24.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.106+0000 7fb1fd2c3640 1 Processor -- start 2024-03-25T19:49:24.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.106+0000 7fb1fd2c3640 1 -- start start 2024-03-25T19:49:24.107 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.106+0000 7fb1fd2c3640 10 monclient: init 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 10 monclient(hunting): picked mon.noname-b con 0x7fb1f80a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 10 monclient(hunting): picked mon.noname-a con 0x7fb1f813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 10 monclient(hunting): picked mon.noname-c con 0x7fb1f81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f8142520 con 0x7fb1f813f710 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f80b28a0 con 0x7fb1f81430c0 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f80b2a30 con 0x7fb1f80a98f0 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:24.108 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.107+0000 7fb1fd2c3640 10 monclient(hunting): authenticate will time out at 5376.376217s 2024-03-25T19:49:24.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.113+0000 7fb1f6ffd640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fb1f80a98f0 0x7fb1f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:44872/0 (socket says 172.21.15.69:44872) 2024-03-25T19:49:24.113 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.113+0000 7fb1f6ffd640 1 -- 172.21.15.69:0/2432302095 learned_addr learned my addr 172.21.15.69:0/2432302095 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:24.128 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.128+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499054024 0 0) 0x7fb1f80b2a30 con 0x7fb1f80a98f0 2024-03-25T19:49:24.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.128+0000 7fb1f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.128+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.128+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.128+0000 7fb1f5ffb640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:49:24.129 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.128+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e0003520 con 0x7fb1f80a98f0 2024-03-25T19:49:24.143 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.143+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049078583 0 0) 0x7fb1f80b28a0 con 0x7fb1f81430c0 2024-03-25T19:49:24.144 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.143+0000 7fb1f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.144 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.143+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.144 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.143+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.144 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.143+0000 7fb1f5ffb640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:49:24.144 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.143+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f80b2a30 con 0x7fb1f81430c0 2024-03-25T19:49:24.146 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.146+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677926029 0 0) 0x7fb1f8142520 con 0x7fb1f813f710 2024-03-25T19:49:24.146 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.146+0000 7fb1f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.146 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.146+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.146 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.146+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.146 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.146+0000 7fb1f5ffb640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:49:24.146 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.146+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f80b28a0 con 0x7fb1f813f710 2024-03-25T19:49:24.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3115372275 0 0) 0x7fb1f80b2a30 con 0x7fb1f81430c0 2024-03-25T19:49:24.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 >> v1:172.21.15.69:6790/0 conn(0x7fb1f80a98f0 legacy=0x7fb1f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 >> v1:172.21.15.69:6789/0 conn(0x7fb1f813f710 legacy=0x7fb1f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.151 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f80b44c0 con 0x7fb1f81430c0 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb1f00019f0 con 0x7fb1f81430c0 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1fd2c3640 5 monclient: authenticate success, global_id 4588 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1fd2c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1f5ffb640 10 monclient: dump: 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:24.152 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:24.153 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.151+0000 7fb1fd2c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:24.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f77fe640 1 -- 172.21.15.69:0/2432302095 >> v1:172.21.15.169:6789/0 conn(0x7fb1f81430c0 legacy=0x7fb1f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:49:24.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f77fe640 1 -- 172.21.15.69:0/2432302095 >> v1:172.21.15.169:6789/0 conn(0x7fb1f81430c0 legacy=0x7fb1f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:49:24.154 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f77fe640 1 --1- 172.21.15.69:0/2432302095 >> v1:172.21.15.169:6789/0 conn(0x7fb1f81430c0 0x7fb1f814dc30 :-1 s=OPENED pgs=64873 cs=1 l=1).handle_message read tag failed 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f77fe640 1 --1- 172.21.15.69:0/2432302095 >> v1:172.21.15.169:6789/0 conn(0x7fb1f81430c0 0x7fb1f814dc30 :-1 s=OPENED pgs=64873 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.169:6789/0 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 >> v1:172.21.15.169:6789/0 conn(0x7fb1f81430c0 legacy=0x7fb1f814dc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 10 monclient(hunting): picked mon.b con 0x7fb1e0003eb0 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 10 monclient(hunting): picked mon.a con 0x7fb1f813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 10 monclient(hunting): picked mon.c con 0x7fb1f80a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb1f80b2a30 con 0x7fb1f813f710 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb1f8142520 con 0x7fb1e0003eb0 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb1e0004590 con 0x7fb1f80a98f0 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:24.155 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.154+0000 7fb1f5ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.169:6789/0 2024-03-25T19:49:24.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.193+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199159819 0 0) 0x7fb1f80b2a30 con 0x7fb1f813f710 2024-03-25T19:49:24.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.193+0000 7fb1f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.193+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.193+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:49:24.193 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.193+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.69:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fb1e000f0c0 con 0x7fb1f813f710 2024-03-25T19:49:24.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.200+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52987756 0 0) 0x7fb1e0004590 con 0x7fb1f80a98f0 2024-03-25T19:49:24.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.200+0000 7fb1f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.200+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.200+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:49:24.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.200+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.69:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fb1f80b2a30 con 0x7fb1f80a98f0 2024-03-25T19:49:24.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.201+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685527038 0 0) 0x7fb1f8142520 con 0x7fb1e0003eb0 2024-03-25T19:49:24.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.201+0000 7fb1f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.201+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.201+0000 7fb1f5ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:49:24.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.201+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.169:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fb1e0004590 con 0x7fb1e0003eb0 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3127878464 0 0) 0x7fb1e000f0c0 con 0x7fb1f813f710 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 >> v1:172.21.15.69:6790/0 conn(0x7fb1f80a98f0 legacy=0x7fb1e000d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 >> v1:172.21.15.169:6789/0 conn(0x7fb1e0003eb0 legacy=0x7fb1e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 1 monclient: found mon.a 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fb1e0012200 con 0x7fb1f813f710 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb1ec001770 con 0x7fb1f813f710 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1f5ffb640 10 monclient: dump: 2024-03-25T19:49:24.216 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:24.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.215+0000 7fb1fd2c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:24.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.227+0000 7fb1f5ffb640 1 -- 172.21.15.69:0/2432302095 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1ec0018d0 con 0x7fb1f813f710 2024-03-25T19:49:24.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.227+0000 7fb1f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:24.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.227+0000 7fb1fd2c3640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:24.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.227+0000 7fb1fd2c3640 4 set_mon_vals no callback set 2024-03-25T19:49:24.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.228+0000 7fb1fd2c3640 10 monclient: shutdown 2024-03-25T19:49:24.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.228+0000 7fb1fd2c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:24.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.228+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/2432302095 >> v1:172.21.15.69:6789/0 conn(0x7fb1f813f710 legacy=0x7fb1e0004ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.228+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/2432302095 shutdown_connections 2024-03-25T19:49:24.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.228+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/2432302095 >> 172.21.15.69:0/2432302095 conn(0x7fb1f80b7ea0 msgr2=0x7fb1f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:24.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.228+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/2432302095 shutdown_connections 2024-03-25T19:49:24.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.228+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/2432302095 wait complete. 2024-03-25T19:49:24.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.229+0000 7fb1fd2c3640 10 monclient: build_initial_monmap 2024-03-25T19:49:24.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.229+0000 7fb1fd2c3640 10 monclient: monmap: 2024-03-25T19:49:24.229 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:24.229 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:24.229 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.229+0000 7fb1fd2c3640 1 librados: starting msgr at 2024-03-25T19:49:24.230 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.229+0000 7fb1fd2c3640 1 librados: starting objecter 2024-03-25T19:49:24.231 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 1 Processor -- start 2024-03-25T19:49:24.231 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 1 -- start start 2024-03-25T19:49:24.231 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 1 librados: setting wanted keys 2024-03-25T19:49:24.231 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 1 librados: calling monclient init 2024-03-25T19:49:24.231 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 10 monclient: init 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e0003eb0 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 10 monclient(hunting): picked mon.noname-a con 0x7fb1f80a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 10 monclient(hunting): picked mon.noname-c con 0x7fb1f813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f813c150 con 0x7fb1f80a98f0 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f813af10 con 0x7fb1e0003eb0 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f81dabe0 con 0x7fb1f813f710 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:24.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.230+0000 7fb1fd2c3640 10 monclient(hunting): authenticate will time out at 5376.499563s 2024-03-25T19:49:24.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.242+0000 7fb1f6ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb1e0003eb0 0x7fb1f81da4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:46050/0 (socket says 172.21.15.69:46050) 2024-03-25T19:49:24.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.242+0000 7fb1f6ffd640 1 -- 172.21.15.69:0/4083039344 learned_addr learned my addr 172.21.15.69:0/4083039344 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:24.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.243+0000 7fb1df7fe640 10 client.?.objecter ms_handle_connect 0x7fb1e0003eb0 2024-03-25T19:49:24.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.243+0000 7fb1df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.250 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.250+0000 7fb1df7fe640 10 client.?.objecter ms_handle_connect 0x7fb1f813f710 2024-03-25T19:49:24.250 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.250+0000 7fb1df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.256+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558640583 0 0) 0x7fb1f813af10 con 0x7fb1e0003eb0 2024-03-25T19:49:24.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.256+0000 7fb1df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.256+0000 7fb1df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.256+0000 7fb1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.256+0000 7fb1df7fe640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:49:24.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.256+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4004eb0 con 0x7fb1e0003eb0 2024-03-25T19:49:24.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.261+0000 7fb1df7fe640 10 client.?.objecter ms_handle_connect 0x7fb1f80a98f0 2024-03-25T19:49:24.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.261+0000 7fb1df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.264+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685754612 0 0) 0x7fb1f81dabe0 con 0x7fb1f813f710 2024-03-25T19:49:24.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.264+0000 7fb1df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.264+0000 7fb1df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.264+0000 7fb1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.264+0000 7fb1df7fe640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:49:24.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.264+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f813af10 con 0x7fb1f813f710 2024-03-25T19:49:24.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.268+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114647449 0 0) 0x7fb1f813c150 con 0x7fb1f80a98f0 2024-03-25T19:49:24.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.268+0000 7fb1df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.268+0000 7fb1df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.268+0000 7fb1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.268+0000 7fb1df7fe640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:49:24.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.268+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f81dabe0 con 0x7fb1f80a98f0 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.283+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2724476173 0 0) 0x7fb1e4004eb0 con 0x7fb1e0003eb0 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.284+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f813c150 con 0x7fb1e0003eb0 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.284+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb1e80018d0 con 0x7fb1e0003eb0 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.284+0000 7fb1df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.284+0000 7fb1df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.284+0000 7fb1df7fe640 10 monclient(hunting): dump: 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:24.284 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:24.285 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:24.285 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:24.285 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.284+0000 7fb1df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.289+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1601579166 0 0) 0x7fb1f813af10 con 0x7fb1f813f710 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.289+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e4004eb0 con 0x7fb1f813f710 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.289+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb1f0001e90 con 0x7fb1f813f710 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.289+0000 7fb1df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.289+0000 7fb1df7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.289+0000 7fb1df7fe640 10 monclient(hunting): dump: 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:24.290 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:24.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.307+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283207542 0 0) 0x7fb1f81dabe0 con 0x7fb1f80a98f0 2024-03-25T19:49:24.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.307+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f813af10 con 0x7fb1f80a98f0 2024-03-25T19:49:24.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.307+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb1ec001450 con 0x7fb1f80a98f0 2024-03-25T19:49:24.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.307+0000 7fb1df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.307+0000 7fb1df7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:24.307 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.307+0000 7fb1df7fe640 10 monclient(hunting): dump: 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:24.308 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1135485570 0 0) 0x7fb1e4004eb0 con 0x7fb1f813f710 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 >> v1:172.21.15.169:6789/0 conn(0x7fb1e0003eb0 legacy=0x7fb1f81da4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 >> v1:172.21.15.69:6789/0 conn(0x7fb1f80a98f0 legacy=0x7fb1f813b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 1 monclient: found mon.c 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f81dad70 con 0x7fb1f813f710 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:24.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1fd2c3640 5 monclient: authenticate success, global_id 4595 2024-03-25T19:49:24.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1fd2c3640 10 monclient: _renew_subs 2024-03-25T19:49:24.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1fd2c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:24.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.309+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f813bc70 con 0x7fb1f813f710 2024-03-25T19:49:24.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.310+0000 7fb1fd2c3640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:24.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.310+0000 7fb1fd2c3640 10 monclient: _renew_subs 2024-03-25T19:49:24.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.310+0000 7fb1fd2c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:24.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.310+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f813bf20 con 0x7fb1f813f710 2024-03-25T19:49:24.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.310+0000 7fb1fd2c3640 1 librados: init done 2024-03-25T19:49:24.312 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.311+0000 7fb1fd2c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.312 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.311+0000 7fb1fd2c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.312 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.311+0000 7fb1fd2c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:24.312 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.311+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f81cd290 con 0x7fb1f813f710 2024-03-25T19:49:24.622 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.620+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f0002030 con 0x7fb1f813f710 2024-03-25T19:49:24.622 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.620+0000 7fb1df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:24.622 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.620+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb1f0003b10 con 0x7fb1f813f710 2024-03-25T19:49:24.622 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.620+0000 7fb1df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.622 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.620+0000 7fb1df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:24.622 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.620+0000 7fb1df7fe640 10 monclient: dump: 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.620+0000 7fb1f57fa640 4 set_mon_vals no callback set 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.621+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fb1f0003cf0 con 0x7fb1f813f710 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.622+0000 7fb1f6ffd640 10 monclient: get_auth_request con 0x7fb1e406e160 auth_method 0 2024-03-25T19:49:24.624 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.622+0000 7fb1df7fe640 10 client.4595.objecter ms_handle_connect 0x7fb1e406e160 2024-03-25T19:49:24.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.627+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fb1f0054900 con 0x7fb1f813f710 2024-03-25T19:49:24.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.627+0000 7fb1df7fe640 10 client.4595.objecter ms_dispatch 0x7fb1f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:24.627 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.627+0000 7fb1df7fe640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:24.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.627+0000 7fb1df7fe640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:24.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.627+0000 7fb1df7fe640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-25T19:49:24.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.627+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1f0050410 con 0x7fb1f813f710 2024-03-25T19:49:24.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.627+0000 7fb1df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.628 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.627+0000 7fb1df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:24.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.885+0000 7fb1fd2c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.885+0000 7fb1fd2c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.885+0000 7fb1fd2c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:24.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.885+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1f80627b0 con 0x7fb1f813f710 2024-03-25T19:49:24.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.926+0000 7fb1df7fe640 1 -- 172.21.15.69:0/4083039344 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fb1f0052340 con 0x7fb1f813f710 2024-03-25T19:49:24.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.926+0000 7fb1df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.926+0000 7fb1df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:24.926 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:49:24.926 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:24.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 10 librados: watch_flush enter 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 10 librados: watch_flush exit 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 10 client.4595.objecter successfully canceled tick 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 >> v1:172.21.15.169:6800/3072548259 conn(0x7fb1e406e160 legacy=0x7fb1e4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 10 monclient: shutdown 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 >> v1:172.21.15.69:6790/0 conn(0x7fb1f813f710 legacy=0x7fb1f81e1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 shutdown_connections 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 >> 172.21.15.69:0/4083039344 conn(0x7fb1f80b7ea0 msgr2=0x7fb1f80b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 shutdown_connections 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 1 -- 172.21.15.69:0/4083039344 wait complete. 2024-03-25T19:49:24.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:24.928+0000 7fb1fd2c3640 1 librados: shutdown 2024-03-25T19:49:24.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:31.746 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:31.864 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.864+0000 7f4071ae2640 10 monclient: get_monmap_and_config 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.864+0000 7f4071ae2640 10 monclient: build_initial_monmap 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.864+0000 7f4071ae2640 10 monclient: monmap: 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:49:31.865624+0000 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:49:31.865624+0000 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:49:31.865 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.865+0000 7f4071ae2640 1 Processor -- start 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.865+0000 7f4071ae2640 1 -- start start 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.865+0000 7f4071ae2640 10 monclient: init 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 10 monclient(hunting): picked mon.noname-b con 0x7f406c0b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 10 monclient(hunting): picked mon.noname-a con 0x7f406c0b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 10 monclient(hunting): picked mon.noname-c con 0x7f406c14be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c0af940 con 0x7f406c0b1730 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c0b0a70 con 0x7f406c14be90 2024-03-25T19:49:31.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c0b0d50 con 0x7f406c0b4d60 2024-03-25T19:49:31.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:31.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.866+0000 7f4071ae2640 10 monclient(hunting): authenticate will time out at 5384.134930s 2024-03-25T19:49:31.873 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.873+0000 7f406b7fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f406c0b4d60 0x7f406c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:32816/0 (socket says 172.21.15.69:32816) 2024-03-25T19:49:31.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.874+0000 7f406b7fe640 1 -- 172.21.15.69:0/1742187167 learned_addr learned my addr 172.21.15.69:0/1742187167 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:31.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.881+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805617630 0 0) 0x7f406c0b0d50 con 0x7f406c0b4d60 2024-03-25T19:49:31.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.881+0000 7f406a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.881+0000 7f406a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.881+0000 7f406a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.881+0000 7f406a7fc640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:49:31.882 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.882+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4054003520 con 0x7f406c0b4d60 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3521093533 0 0) 0x7f4054003520 con 0x7f406c0b4d60 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 >> v1:172.21.15.169:6789/0 conn(0x7f406c14be90 legacy=0x7f406c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 >> v1:172.21.15.69:6789/0 conn(0x7f406c0b1730 legacy=0x7f406c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0b1030 con 0x7f406c0b4d60 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f405c001690 con 0x7f406c0b4d60 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f406a7fc640 10 monclient: dump: 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f4071ae2640 5 monclient: authenticate success, global_id 4598 2024-03-25T19:49:31.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.893+0000 7f4071ae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.903+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f405c0017f0 con 0x7f406c0b4d60 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f406a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f406a7fc640 1 -- 172.21.15.69:0/1742187167 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f405c001c00 con 0x7f406c0b4d60 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f406a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f406a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f406a7fc640 10 monclient: dump: 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:31.904 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f4071ae2640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:31.905 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f4071ae2640 4 set_mon_vals no callback set 2024-03-25T19:49:31.907 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f4071ae2640 10 monclient: shutdown 2024-03-25T19:49:31.907 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f4071ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:31.907 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f4071ae2640 1 -- 172.21.15.69:0/1742187167 >> v1:172.21.15.69:6790/0 conn(0x7f406c0b4d60 legacy=0x7f406c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.907 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f4071ae2640 1 -- 172.21.15.69:0/1742187167 shutdown_connections 2024-03-25T19:49:31.907 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f4071ae2640 1 -- 172.21.15.69:0/1742187167 >> 172.21.15.69:0/1742187167 conn(0x7f406c0617a0 msgr2=0x7f406c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.904+0000 7f4071ae2640 1 -- 172.21.15.69:0/1742187167 shutdown_connections 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.905+0000 7f4071ae2640 1 -- 172.21.15.69:0/1742187167 wait complete. 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.905+0000 7f4071ae2640 10 monclient: build_initial_monmap 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.905+0000 7f4071ae2640 10 monclient: monmap: 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:31.908 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.905+0000 7f4071ae2640 1 librados: starting msgr at 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.905+0000 7f4071ae2640 1 librados: starting objecter 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.905+0000 7f4071ae2640 1 Processor -- start 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 1 -- start start 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 1 librados: setting wanted keys 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 1 librados: calling monclient init 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 10 monclient: init 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 10 monclient(hunting): picked mon.noname-c con 0x7f406c0b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 10 monclient(hunting): picked mon.noname-b con 0x7f406c0b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 10 monclient(hunting): picked mon.noname-a con 0x7f406c14be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:31.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c13e050 con 0x7f406c14be90 2024-03-25T19:49:31.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c1452a0 con 0x7f406c0b4d60 2024-03-25T19:49:31.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c13db70 con 0x7f406c0b1730 2024-03-25T19:49:31.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:31.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.906+0000 7f4071ae2640 10 monclient(hunting): authenticate will time out at 5384.175444s 2024-03-25T19:49:31.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.916+0000 7f406b7fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f406c0b1730 0x7f406c1d78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:32850/0 (socket says 172.21.15.69:32850) 2024-03-25T19:49:31.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.916+0000 7f406b7fe640 1 -- 172.21.15.69:0/716032510 learned_addr learned my addr 172.21.15.69:0/716032510 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:31.917 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.917+0000 7f4053fff640 10 client.?.objecter ms_handle_connect 0x7f406c0b1730 2024-03-25T19:49:31.917 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.917+0000 7f4053fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.921+0000 7f4053fff640 10 client.?.objecter ms_handle_connect 0x7f406c0b4d60 2024-03-25T19:49:31.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.921+0000 7f4053fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.933+0000 7f4053fff640 10 client.?.objecter ms_handle_connect 0x7f406c14be90 2024-03-25T19:49:31.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.933+0000 7f4053fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.936+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295523944 0 0) 0x7f406c13db70 con 0x7f406c0b1730 2024-03-25T19:49:31.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.936+0000 7f4053fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.936+0000 7f4053fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.936+0000 7f4053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.936+0000 7f4053fff640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:49:31.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.936+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4058004eb0 con 0x7f406c0b1730 2024-03-25T19:49:31.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.937+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661025044 0 0) 0x7f406c1452a0 con 0x7f406c0b4d60 2024-03-25T19:49:31.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.937+0000 7f4053fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.937+0000 7f4053fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.937+0000 7f4053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.937+0000 7f4053fff640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:49:31.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.937+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c13db70 con 0x7f406c0b4d60 2024-03-25T19:49:31.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.943+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889371844 0 0) 0x7f406c13e050 con 0x7f406c14be90 2024-03-25T19:49:31.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.943+0000 7f4053fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.943+0000 7f4053fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.943+0000 7f4053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.943+0000 7f4053fff640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:49:31.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.943+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c1452a0 con 0x7f406c14be90 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.966+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4201651982 0 0) 0x7f406c1452a0 con 0x7f406c14be90 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.966+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f406c13e050 con 0x7f406c14be90 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.966+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4064001a60 con 0x7f406c14be90 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.966+0000 7f4053fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.967+0000 7f4053fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.967+0000 7f4053fff640 10 monclient(hunting): dump: 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.967 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.968 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.968 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:31.968 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:31.968 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:31.968 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:31.968 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:31.968 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.967+0000 7f4053fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:31.972 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.972+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1422800517 0 0) 0x7f406c13db70 con 0x7f406c0b4d60 2024-03-25T19:49:31.972 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.972+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f406c1452a0 con 0x7f406c0b4d60 2024-03-25T19:49:31.972 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.972+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f40600018b0 con 0x7f406c0b4d60 2024-03-25T19:49:31.972 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.972+0000 7f4053fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.972+0000 7f4053fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.972+0000 7f4053fff640 10 monclient(hunting): dump: 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:31.973 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.975+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 84431769 0 0) 0x7f4058004eb0 con 0x7f406c0b1730 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.975+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f406c13db70 con 0x7f406c0b1730 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.975+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f405c001c00 con 0x7f406c0b1730 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.975+0000 7f4053fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.975+0000 7f4053fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.975+0000 7f4053fff640 10 monclient(hunting): dump: 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:31.975 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.976 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:31.976 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:31.976 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:31.976 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:31.976 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:31.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1407570164 0 0) 0x7f406c13e050 con 0x7f406c14be90 2024-03-25T19:49:31.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 >> v1:172.21.15.69:6790/0 conn(0x7f406c0b1730 legacy=0x7f406c1d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 >> v1:172.21.15.169:6789/0 conn(0x7f406c0b4d60 legacy=0x7f406c1d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:31.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 1 monclient: found mon.a 2024-03-25T19:49:31.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:31.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c13dd00 con 0x7f406c14be90 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4071ae2640 5 monclient: authenticate success, global_id 4629 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4071ae2640 10 monclient: _renew_subs 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4071ae2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.986+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c1455a0 con 0x7f406c14be90 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.987+0000 7f4071ae2640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.987+0000 7f4071ae2640 10 monclient: _renew_subs 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.987+0000 7f4071ae2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.987+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c135e80 con 0x7f406c14be90 2024-03-25T19:49:31.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.987+0000 7f4071ae2640 1 librados: init done 2024-03-25T19:49:31.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.988+0000 7f4071ae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.988+0000 7f4071ae2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.988+0000 7f4071ae2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:49:31.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:31.988+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c0627b0 con 0x7f406c14be90 2024-03-25T19:49:32.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.297+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4064001bc0 con 0x7f406c14be90 2024-03-25T19:49:32.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.297+0000 7f4053fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:32.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.297+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4064003b10 con 0x7f406c14be90 2024-03-25T19:49:32.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.297+0000 7f4053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.297+0000 7f4053fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.297+0000 7f4053fff640 10 monclient: dump: 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:32.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.297+0000 7f4069ffb640 4 set_mon_vals no callback set 2024-03-25T19:49:32.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.297+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f4064019ae0 con 0x7f406c14be90 2024-03-25T19:49:32.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.298+0000 7f406b7fe640 10 monclient: get_auth_request con 0x7f405806e060 auth_method 0 2024-03-25T19:49:32.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.299+0000 7f4053fff640 10 client.4629.objecter ms_handle_connect 0x7f405806e060 2024-03-25T19:49:32.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.304+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f4064053710 con 0x7f406c14be90 2024-03-25T19:49:32.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.304+0000 7f4053fff640 10 client.4629.objecter ms_dispatch 0x7f406c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:32.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.304+0000 7f4053fff640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:32.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.304+0000 7f4053fff640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:32.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.304+0000 7f4053fff640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T19:49:32.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.304+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f406404f050 con 0x7f406c14be90 2024-03-25T19:49:32.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.304+0000 7f4053fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:32.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.304+0000 7f4053fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:32.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.567+0000 7f4071ae2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.567+0000 7f4071ae2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.567+0000 7f4071ae2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:49:32.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.567+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f406c13e200 con 0x7f406c14be90 2024-03-25T19:49:32.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.598+0000 7f4053fff640 1 -- 172.21.15.69:0/716032510 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f406c13e200 con 0x7f406c14be90 2024-03-25T19:49:32.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.598+0000 7f4053fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.598+0000 7f4053fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:32.599 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:49:32.599 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:32.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 10 librados: watch_flush enter 2024-03-25T19:49:32.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 10 librados: watch_flush exit 2024-03-25T19:49:32.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T19:49:32.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 10 client.4629.objecter successfully canceled tick 2024-03-25T19:49:32.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 >> v1:172.21.15.169:6800/3072548259 conn(0x7f405806e060 legacy=0x7f4058070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:32.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 10 monclient: shutdown 2024-03-25T19:49:32.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:32.602 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 >> v1:172.21.15.69:6789/0 conn(0x7f406c14be90 legacy=0x7f406c1e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:32.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 shutdown_connections 2024-03-25T19:49:32.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 >> 172.21.15.69:0/716032510 conn(0x7f406c0617a0 msgr2=0x7f406c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:32.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 shutdown_connections 2024-03-25T19:49:32.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 1 -- 172.21.15.69:0/716032510 wait complete. 2024-03-25T19:49:32.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:32.601+0000 7f4071ae2640 1 librados: shutdown 2024-03-25T19:49:32.614 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:33.483 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:11.446 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.533 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:11.497 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.581 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:11.545 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.593 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:11.547 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.601 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:11.555 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.613 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:11.577 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.617 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:11.581 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.629 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:11.583 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.636 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:11.600 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:39.420 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:39.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.529+0000 7f52b2416640 10 monclient: get_monmap_and_config 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.529+0000 7f52b2416640 10 monclient: build_initial_monmap 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.529+0000 7f52b2416640 10 monclient: monmap: 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:49:39.531127+0000 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:49:39.531127+0000 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.530+0000 7f52b2416640 1 Processor -- start 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.530+0000 7f52b2416640 1 -- start start 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.530+0000 7f52b2416640 10 monclient: init 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 10 monclient(hunting): picked mon.noname-b con 0x7f52ac0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 10 monclient(hunting): picked mon.noname-a con 0x7f52ac13f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 10 monclient(hunting): picked mon.noname-c con 0x7f52ac1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac142520 con 0x7f52ac13f710 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac0b28a0 con 0x7f52ac1430c0 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac0b2a30 con 0x7f52ac0a98f0 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:39.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.531+0000 7f52b2416640 10 monclient(hunting): authenticate will time out at 5391.799929s 2024-03-25T19:49:39.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.570+0000 7f52abfff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f52ac0a98f0 0x7f52ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:37480/0 (socket says 172.21.15.69:37480) 2024-03-25T19:49:39.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.570+0000 7f52abfff640 1 -- 172.21.15.69:0/1120966696 learned_addr learned my addr 172.21.15.69:0/1120966696 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:39.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.589+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576270266 0 0) 0x7f52ac0b2a30 con 0x7f52ac0a98f0 2024-03-25T19:49:39.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.589+0000 7f52aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.589+0000 7f52aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.589+0000 7f52aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.589+0000 7f52aaffd640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:49:39.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.589+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5294003520 con 0x7f52ac0a98f0 2024-03-25T19:49:39.592 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.591+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250952244 0 0) 0x7f52ac0b28a0 con 0x7f52ac1430c0 2024-03-25T19:49:39.592 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.591+0000 7f52aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.592 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.591+0000 7f52aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.592 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.591+0000 7f52aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.592 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.591+0000 7f52aaffd640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:49:39.592 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.591+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ac0b2a30 con 0x7f52ac1430c0 2024-03-25T19:49:39.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4125672656 0 0) 0x7f5294003520 con 0x7f52ac0a98f0 2024-03-25T19:49:39.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 >> v1:172.21.15.169:6789/0 conn(0x7f52ac1430c0 legacy=0x7f52ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 >> v1:172.21.15.69:6789/0 conn(0x7f52ac13f710 legacy=0x7f52ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 1 monclient: found mon.noname-b 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac0b44c0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f529c001690 con 0x7f52ac0a98f0 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52b2416640 5 monclient: authenticate success, global_id 4604 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52b2416640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:39.604 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52aaffd640 10 monclient: dump: 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:39.605 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.603+0000 7f52b2416640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:39.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.611+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f529c0017f0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.611+0000 7f52aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:39.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.611+0000 7f52b2416640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:39.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.611+0000 7f52b2416640 4 set_mon_vals no callback set 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.611+0000 7f52aaffd640 1 -- 172.21.15.69:0/1120966696 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f529c001c00 con 0x7f52ac0a98f0 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.611+0000 7f52aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52aaffd640 10 monclient: dump: 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:39.613 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52b2416640 10 monclient: shutdown 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52b2416640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52b2416640 1 -- 172.21.15.69:0/1120966696 >> v1:172.21.15.69:6790/0 conn(0x7f52ac0a98f0 legacy=0x7f52ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52b2416640 1 -- 172.21.15.69:0/1120966696 shutdown_connections 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52b2416640 1 -- 172.21.15.69:0/1120966696 >> 172.21.15.69:0/1120966696 conn(0x7f52ac0b7ea0 msgr2=0x7f52ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52b2416640 1 -- 172.21.15.69:0/1120966696 shutdown_connections 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.612+0000 7f52b2416640 1 -- 172.21.15.69:0/1120966696 wait complete. 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 10 monclient: build_initial_monmap 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 10 monclient: monmap: 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:39.614 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 1 librados: starting msgr at 2024-03-25T19:49:39.615 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 1 librados: starting objecter 2024-03-25T19:49:39.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 1 Processor -- start 2024-03-25T19:49:39.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 1 -- start start 2024-03-25T19:49:39.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 1 librados: setting wanted keys 2024-03-25T19:49:39.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 1 librados: calling monclient init 2024-03-25T19:49:39.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.613+0000 7f52b2416640 10 monclient: init 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 10 monclient(hunting): picked mon.noname-c con 0x7f52ac0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 10 monclient(hunting): picked mon.noname-b con 0x7f52ac13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 10 monclient(hunting): picked mon.noname-a con 0x7f52ac1430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac1d51f0 con 0x7f52ac1430c0 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac1dac30 con 0x7f52ac13f710 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac0ab9e0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:39.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.614+0000 7f52b2416640 10 monclient(hunting): authenticate will time out at 5391.883139s 2024-03-25T19:49:39.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.619+0000 7f52abfff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f52ac0a98f0 0x7f52ac1382c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:37502/0 (socket says 172.21.15.69:37502) 2024-03-25T19:49:39.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.619+0000 7f52abfff640 1 -- 172.21.15.69:0/1142663989 learned_addr learned my addr 172.21.15.69:0/1142663989 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:39.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.623+0000 7f52a8ff9640 10 client.?.objecter ms_handle_connect 0x7f52ac0a98f0 2024-03-25T19:49:39.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.623+0000 7f52a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.630+0000 7f52a8ff9640 10 client.?.objecter ms_handle_connect 0x7f52ac13f710 2024-03-25T19:49:39.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.630+0000 7f52a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.632+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135003624 0 0) 0x7f52ac0ab9e0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.632+0000 7f52a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.632+0000 7f52a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.632+0000 7f52a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.632+0000 7f52a8ff9640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:49:39.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.632+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5298004eb0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.634+0000 7f52a8ff9640 10 client.?.objecter ms_handle_connect 0x7f52ac1430c0 2024-03-25T19:49:39.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.634+0000 7f52a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.647+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622758121 0 0) 0x7f52ac1dac30 con 0x7f52ac13f710 2024-03-25T19:49:39.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.647+0000 7f52a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.647+0000 7f52a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.647+0000 7f52a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.647+0000 7f52a8ff9640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:49:39.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.647+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ac0ab9e0 con 0x7f52ac13f710 2024-03-25T19:49:39.649 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.649+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916017516 0 0) 0x7f52ac1d51f0 con 0x7f52ac1430c0 2024-03-25T19:49:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.649+0000 7f52a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.649+0000 7f52a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.649+0000 7f52a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.649+0000 7f52a8ff9640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:49:39.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.649+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ac1dac30 con 0x7f52ac1430c0 2024-03-25T19:49:39.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.658+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 510653238 0 0) 0x7f5298004eb0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.658+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52ac1d51f0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.658+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f529c001c00 con 0x7f52ac0a98f0 2024-03-25T19:49:39.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.658+0000 7f52a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.658+0000 7f52a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:39.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.658+0000 7f52a8ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:39.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.658+0000 7f52a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:39.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.664+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 953010530 0 0) 0x7f52ac0ab9e0 con 0x7f52ac13f710 2024-03-25T19:49:39.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.664+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5298004eb0 con 0x7f52ac13f710 2024-03-25T19:49:39.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.664+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f52a00018a0 con 0x7f52ac13f710 2024-03-25T19:49:39.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.664+0000 7f52a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.664+0000 7f52a8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:39.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.664+0000 7f52a8ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:39.664 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:39.664 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2557958646 0 0) 0x7f52ac1d51f0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 >> v1:172.21.15.169:6789/0 conn(0x7f52ac13f710 legacy=0x7f52ac1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 >> v1:172.21.15.69:6789/0 conn(0x7f52ac1430c0 legacy=0x7f52ac1e1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 1 monclient: found mon.c 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac1389d0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 5 monclient: authenticate success, global_id 4607 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 10 monclient: _renew_subs 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ac1db860 con 0x7f52ac0a98f0 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 10 monclient: _renew_subs 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ac138e20 con 0x7f52ac0a98f0 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.666+0000 7f52b2416640 1 librados: init done 2024-03-25T19:49:39.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.667+0000 7f52b2416640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.667+0000 7f52b2416640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.667+0000 7f52b2416640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:39.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.667+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ac1dadc0 con 0x7f52ac0a98f0 2024-03-25T19:49:39.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.973+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f529c004a90 con 0x7f52ac0a98f0 2024-03-25T19:49:39.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.973+0000 7f52a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:39.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.973+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f529c004e20 con 0x7f52ac0a98f0 2024-03-25T19:49:39.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.973+0000 7f52a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.973+0000 7f52a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:39.973 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.973+0000 7f52a8ff9640 10 monclient: dump: 2024-03-25T19:49:39.973 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:39.974 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.973+0000 7f52aa7fc640 4 set_mon_vals no callback set 2024-03-25T19:49:39.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.973+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f529c005000 con 0x7f52ac0a98f0 2024-03-25T19:49:39.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.974+0000 7f52ab7fe640 10 monclient: get_auth_request con 0x7f529806df70 auth_method 0 2024-03-25T19:49:39.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.974+0000 7f52a8ff9640 10 client.4607.objecter ms_handle_connect 0x7f529806df70 2024-03-25T19:49:39.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.979+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f529c054980 con 0x7f52ac0a98f0 2024-03-25T19:49:39.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.979+0000 7f52a8ff9640 10 client.4607.objecter ms_dispatch 0x7f52ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:39.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.979+0000 7f52a8ff9640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:39.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.979+0000 7f52a8ff9640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:39.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.979+0000 7f52a8ff9640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-25T19:49:39.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.980+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f529c050490 con 0x7f52ac0a98f0 2024-03-25T19:49:39.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.980+0000 7f52a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:39.980+0000 7f52a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:40.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.236+0000 7f52b2416640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:40.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.236+0000 7f52b2416640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:40.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.236+0000 7f52b2416640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:40.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.236+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52ac0627b0 con 0x7f52ac0a98f0 2024-03-25T19:49:40.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.268+0000 7f52a8ff9640 1 -- 172.21.15.69:0/1142663989 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f529c0523c0 con 0x7f52ac0a98f0 2024-03-25T19:49:40.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.268+0000 7f52a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:40.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.268+0000 7f52a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:40.268 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:49:40.269 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 10 librados: watch_flush enter 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 10 librados: watch_flush exit 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 10 client.4607.objecter successfully canceled tick 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 >> v1:172.21.15.169:6800/3072548259 conn(0x7f529806df70 legacy=0x7f5298070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 10 monclient: shutdown 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 >> v1:172.21.15.69:6790/0 conn(0x7f52ac0a98f0 legacy=0x7f52ac1382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 shutdown_connections 2024-03-25T19:49:40.271 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.270+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 >> 172.21.15.69:0/1142663989 conn(0x7f52ac0b7ea0 msgr2=0x7f52ac13ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:40.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.271+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 shutdown_connections 2024-03-25T19:49:40.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.271+0000 7f52b2416640 1 -- 172.21.15.69:0/1142663989 wait complete. 2024-03-25T19:49:40.272 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:40.271+0000 7f52b2416640 1 librados: shutdown 2024-03-25T19:49:40.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:47.088 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.201+0000 7f4cc7293640 10 monclient: get_monmap_and_config 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.201+0000 7f4cc7293640 10 monclient: build_initial_monmap 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.201+0000 7f4cc7293640 10 monclient: monmap: 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:49:47.202324+0000 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:49:47.202324+0000 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:47.201 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:47.202 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:47.202 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:49:47.202 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:49:47.202 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.201+0000 7f4cc7293640 1 Processor -- start 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.201+0000 7f4cc7293640 1 -- start start 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 10 monclient: init 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc00a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc013f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc0142520 con 0x7f4cc013f710 2024-03-25T19:49:47.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc00b28a0 con 0x7f4cc01430c0 2024-03-25T19:49:47.204 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc00b2a30 con 0x7f4cc00a98f0 2024-03-25T19:49:47.204 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:47.204 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.202+0000 7f4cc7293640 10 monclient(hunting): authenticate will time out at 5399.471274s 2024-03-25T19:49:47.208 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.208+0000 7f4cc482e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4cc013f710 0x7f4cc01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54570/0 (socket says 172.21.15.69:54570) 2024-03-25T19:49:47.208 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.208+0000 7f4cc482e640 1 -- 172.21.15.69:0/1420173250 learned_addr learned my addr 172.21.15.69:0/1420173250 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:47.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.221+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799811798 0 0) 0x7f4cc0142520 con 0x7f4cc013f710 2024-03-25T19:49:47.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.221+0000 7f4cb7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.221+0000 7f4cb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.221+0000 7f4cb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.221+0000 7f4cb7fff640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:49:47.221 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.221+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cac003520 con 0x7f4cc013f710 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.230+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301532901 0 0) 0x7f4cc00b2a30 con 0x7f4cc00a98f0 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.230+0000 7f4cb7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.230+0000 7f4cb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.230+0000 7f4cb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.230+0000 7f4cb7fff640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.230+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc0142520 con 0x7f4cc00a98f0 2024-03-25T19:49:47.231 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.231+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777189306 0 0) 0x7f4cc00b28a0 con 0x7f4cc01430c0 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.231+0000 7f4cb7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.231+0000 7f4cb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.231+0000 7f4cb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.231+0000 7f4cb7fff640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.231+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc00b2a30 con 0x7f4cc01430c0 2024-03-25T19:49:47.238 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679420387 0 0) 0x7f4cac003520 con 0x7f4cc013f710 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 >> v1:172.21.15.69:6790/0 conn(0x7f4cc00a98f0 legacy=0x7f4cc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 >> v1:172.21.15.169:6789/0 conn(0x7f4cc01430c0 legacy=0x7f4cc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 1 monclient: found mon.noname-a 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc00b44c0 con 0x7f4cc013f710 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4cb0001690 con 0x7f4cc013f710 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cb7fff640 10 monclient: dump: 2024-03-25T19:49:47.239 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cc7293640 5 monclient: authenticate success, global_id 4638 2024-03-25T19:49:47.240 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.238+0000 7f4cc7293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:47.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.262+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cb00017f0 con 0x7f4cc013f710 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.262+0000 7f4cb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.262+0000 7f4cb7fff640 1 -- 172.21.15.69:0/1420173250 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4cb0001c00 con 0x7f4cc013f710 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.262+0000 7f4cc7293640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.262+0000 7f4cc7293640 4 set_mon_vals no callback set 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.262+0000 7f4cb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.262+0000 7f4cb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.262+0000 7f4cb7fff640 10 monclient: dump: 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:47.263 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:47.264 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:47.264 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.263+0000 7f4cc7293640 10 monclient: shutdown 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.263+0000 7f4cc7293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.263+0000 7f4cc7293640 1 -- 172.21.15.69:0/1420173250 >> v1:172.21.15.69:6789/0 conn(0x7f4cc013f710 legacy=0x7f4cc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.263+0000 7f4cc7293640 1 -- 172.21.15.69:0/1420173250 shutdown_connections 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.263+0000 7f4cc7293640 1 -- 172.21.15.69:0/1420173250 >> 172.21.15.69:0/1420173250 conn(0x7f4cc00b7ea0 msgr2=0x7f4cc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.263+0000 7f4cc7293640 1 -- 172.21.15.69:0/1420173250 shutdown_connections 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.263+0000 7f4cc7293640 1 -- 172.21.15.69:0/1420173250 wait complete. 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.264+0000 7f4cc7293640 10 monclient: build_initial_monmap 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.264+0000 7f4cc7293640 10 monclient: monmap: 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:47.265 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 1 librados: starting msgr at 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 1 librados: starting objecter 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 1 Processor -- start 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 1 -- start start 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 1 librados: setting wanted keys 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 1 librados: calling monclient init 2024-03-25T19:49:47.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 10 monclient: init 2024-03-25T19:49:47.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:47.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc00a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.265+0000 7f4cc7293640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc013f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.266+0000 7f4cc7293640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc01430c0 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.266+0000 7f4cc7293640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc0137010 con 0x7f4cc01430c0 2024-03-25T19:49:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.266+0000 7f4cc7293640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc0135100 con 0x7f4cc00a98f0 2024-03-25T19:49:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.266+0000 7f4cc7293640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc0135290 con 0x7f4cc013f710 2024-03-25T19:49:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.266+0000 7f4cc7293640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:47.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.266+0000 7f4cc7293640 10 monclient(hunting): authenticate will time out at 5399.534665s 2024-03-25T19:49:47.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.274+0000 7f4cc482e640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f4cc013f710 0x7f4cc01dab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:42244/0 (socket says 172.21.15.69:42244) 2024-03-25T19:49:47.275 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.274+0000 7f4cc482e640 1 -- 172.21.15.69:0/2215575352 learned_addr learned my addr 172.21.15.69:0/2215575352 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:47.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.281+0000 7f4cb5ffb640 10 client.?.objecter ms_handle_connect 0x7f4cc013f710 2024-03-25T19:49:47.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.281+0000 7f4cb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:47.286 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.286+0000 7f4cb5ffb640 10 client.?.objecter ms_handle_connect 0x7f4cc01430c0 2024-03-25T19:49:47.286 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.286+0000 7f4cb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:47.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.289+0000 7f4cb5ffb640 10 client.?.objecter ms_handle_connect 0x7f4cc00a98f0 2024-03-25T19:49:47.290 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.289+0000 7f4cb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:47.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.293+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081446885 0 0) 0x7f4cc0135290 con 0x7f4cc013f710 2024-03-25T19:49:47.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.293+0000 7f4cb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.293+0000 7f4cb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.293+0000 7f4cb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.293+0000 7f4cb5ffb640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:49:47.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.293+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca4004eb0 con 0x7f4cc013f710 2024-03-25T19:49:47.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.295+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121742178 0 0) 0x7f4cc0137010 con 0x7f4cc01430c0 2024-03-25T19:49:47.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.295+0000 7f4cb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.295+0000 7f4cb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.295+0000 7f4cb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.295+0000 7f4cb5ffb640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:49:47.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.295+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc0135290 con 0x7f4cc01430c0 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.309+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239427282 0 0) 0x7f4ca4004eb0 con 0x7f4cc013f710 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.309+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cc0137010 con 0x7f4cc013f710 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.309+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4cb0001bc0 con 0x7f4cc013f710 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.309+0000 7f4cb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.309+0000 7f4cb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.309+0000 7f4cb5ffb640 10 monclient(hunting): dump: 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:47.310 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:47.311 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:47.311 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:47.311 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:47.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.309+0000 7f4cb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:47.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.314+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082964761 0 0) 0x7f4cc0135100 con 0x7f4cc00a98f0 2024-03-25T19:49:47.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.314+0000 7f4cb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.314+0000 7f4cb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.314+0000 7f4cb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.314+0000 7f4cb5ffb640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:49:47.315 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.314+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca4004eb0 con 0x7f4cc00a98f0 2024-03-25T19:49:47.323 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.323+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1596392503 0 0) 0x7f4cc0135290 con 0x7f4cc01430c0 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.323+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cc0135100 con 0x7f4cc01430c0 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.323+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4cb8001b50 con 0x7f4cc01430c0 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.323+0000 7f4cb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.323+0000 7f4cb5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.323+0000 7f4cb5ffb640 10 monclient(hunting): dump: 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:47.324 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:47.325 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:47.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2065416836 0 0) 0x7f4cc0137010 con 0x7f4cc013f710 2024-03-25T19:49:47.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 >> v1:172.21.15.169:6789/0 conn(0x7f4cc00a98f0 legacy=0x7f4cc0130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 >> v1:172.21.15.69:6789/0 conn(0x7f4cc01430c0 legacy=0x7f4cc01e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 1 monclient: found mon.c 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc0135550 con 0x7f4cc013f710 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cc7293640 5 monclient: authenticate success, global_id 4613 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cc7293640 10 monclient: _renew_subs 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cc7293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.325+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc0135820 con 0x7f4cc013f710 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.326+0000 7f4cc7293640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.326+0000 7f4cc7293640 10 monclient: _renew_subs 2024-03-25T19:49:47.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.326+0000 7f4cc7293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:47.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.326+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc013b800 con 0x7f4cc013f710 2024-03-25T19:49:47.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.326+0000 7f4cc7293640 1 librados: init done 2024-03-25T19:49:47.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.327+0000 7f4cc7293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.327+0000 7f4cc7293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.327+0000 7f4cc7293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:47.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.327+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc00b7ac0 con 0x7f4cc013f710 2024-03-25T19:49:47.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.629+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cb00011f0 con 0x7f4cc013f710 2024-03-25T19:49:47.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.629+0000 7f4cb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:47.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.629+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4cb0004c30 con 0x7f4cc013f710 2024-03-25T19:49:47.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.629+0000 7f4cb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.629+0000 7f4cb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.629+0000 7f4cb5ffb640 10 monclient: dump: 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:47.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.629+0000 7f4cb77fe640 4 set_mon_vals no callback set 2024-03-25T19:49:47.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.630+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f4cb001ac80 con 0x7f4cc013f710 2024-03-25T19:49:47.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.631+0000 7f4cc502f640 10 monclient: get_auth_request con 0x7f4ca406df70 auth_method 0 2024-03-25T19:49:47.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.631+0000 7f4cb5ffb640 10 client.4613.objecter ms_handle_connect 0x7f4ca406df70 2024-03-25T19:49:47.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.635+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f4cb0054990 con 0x7f4cc013f710 2024-03-25T19:49:47.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.635+0000 7f4cb5ffb640 10 client.4613.objecter ms_dispatch 0x7f4cc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:47.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.635+0000 7f4cb5ffb640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:47.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.635+0000 7f4cb5ffb640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:47.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.636+0000 7f4cb5ffb640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-25T19:49:47.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.636+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4cb00502d0 con 0x7f4cc013f710 2024-03-25T19:49:47.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.636+0000 7f4cb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.636 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.636+0000 7f4cb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:47.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.868+0000 7f4cc7293640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.868+0000 7f4cc7293640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.868+0000 7f4cc7293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:49:47.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.868+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cc0074cc0 con 0x7f4cc013f710 2024-03-25T19:49:47.906 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.905+0000 7f4cb5ffb640 1 -- 172.21.15.69:0/2215575352 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f4cb00523d0 con 0x7f4cc013f710 2024-03-25T19:49:47.906 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.905+0000 7f4cb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.906 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.905+0000 7f4cb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:47.906 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:49:47.906 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:47.908 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 10 librados: watch_flush enter 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 10 librados: watch_flush exit 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 10 client.4613.objecter successfully canceled tick 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 >> v1:172.21.15.169:6800/3072548259 conn(0x7f4ca406df70 legacy=0x7f4ca4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 10 monclient: shutdown 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 >> v1:172.21.15.69:6790/0 conn(0x7f4cc013f710 legacy=0x7f4cc01dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 shutdown_connections 2024-03-25T19:49:47.909 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.908+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 >> 172.21.15.69:0/2215575352 conn(0x7f4cc00b7ea0 msgr2=0x7f4cc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:47.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.909+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 shutdown_connections 2024-03-25T19:49:47.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.909+0000 7f4cc7293640 1 -- 172.21.15.69:0/2215575352 wait complete. 2024-03-25T19:49:47.910 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:47.909+0000 7f4cc7293640 1 librados: shutdown 2024-03-25T19:49:47.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:53.610 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:31.574 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.617 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:31.581 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.651 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:31.614 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.665 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:31.629 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.676 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:31.640 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.682 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:31.636 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.693 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:31.647 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.697 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:31.661 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.720 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:31.674 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:54.730 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:54.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.836+0000 7eff45979640 10 monclient: get_monmap_and_config 2024-03-25T19:49:54.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.836+0000 7eff45979640 10 monclient: build_initial_monmap 2024-03-25T19:49:54.836 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.836+0000 7eff45979640 10 monclient: monmap: 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:49:54.837988+0000 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:49:54.837988+0000 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:49:54.837 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:54.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 1 Processor -- start 2024-03-25T19:49:54.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 1 -- start start 2024-03-25T19:49:54.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 10 monclient: init 2024-03-25T19:49:54.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:54.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:54.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 10 monclient(hunting): picked mon.noname-b con 0x7eff400b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:54.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 10 monclient(hunting): picked mon.noname-c con 0x7eff400b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:54.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 10 monclient(hunting): picked mon.noname-a con 0x7eff4014be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:54.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff400af940 con 0x7eff4014be90 2024-03-25T19:49:54.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff400b0a70 con 0x7eff400b1730 2024-03-25T19:49:54.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff400b0d50 con 0x7eff400b4d60 2024-03-25T19:49:54.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:54.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.837+0000 7eff45979640 10 monclient(hunting): authenticate will time out at 5407.106584s 2024-03-25T19:49:54.861 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.861+0000 7eff3e7fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7eff400b1730 0x7eff400b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:56570/0 (socket says 172.21.15.69:56570) 2024-03-25T19:49:54.861 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.861+0000 7eff3e7fc640 1 -- 172.21.15.69:0/2515635839 learned_addr learned my addr 172.21.15.69:0/2515635839 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.884+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512046849 0 0) 0x7eff400af940 con 0x7eff4014be90 2024-03-25T19:49:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.884+0000 7eff3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.884+0000 7eff3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.884+0000 7eff3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.884+0000 7eff3dffb640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:49:54.885 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.884+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff28003520 con 0x7eff4014be90 2024-03-25T19:49:54.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.890+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914357685 0 0) 0x7eff400b0a70 con 0x7eff400b1730 2024-03-25T19:49:54.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.890+0000 7eff3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.890+0000 7eff3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.890+0000 7eff3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.890+0000 7eff3dffb640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:49:54.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.890+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff400af940 con 0x7eff400b1730 2024-03-25T19:49:54.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.898+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122922248 0 0) 0x7eff400b0d50 con 0x7eff400b4d60 2024-03-25T19:49:54.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.898+0000 7eff3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.898+0000 7eff3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.898+0000 7eff3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.898+0000 7eff3dffb640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:49:54.898 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.898+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff400b0a70 con 0x7eff400b4d60 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.900+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 224931236 0 0) 0x7eff28003520 con 0x7eff4014be90 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.900+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 >> v1:172.21.15.69:6790/0 conn(0x7eff400b4d60 legacy=0x7eff4013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 >> v1:172.21.15.169:6789/0 conn(0x7eff400b1730 legacy=0x7eff400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 1 monclient: found mon.noname-a 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff400b1030 con 0x7eff4014be90 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7eff38001a10 con 0x7eff4014be90 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff3dffb640 10 monclient: dump: 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff45979640 5 monclient: authenticate success, global_id 4644 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.901+0000 7eff45979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.977+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff38001bf0 con 0x7eff4014be90 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.977+0000 7eff3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.977+0000 7eff3dffb640 1 -- 172.21.15.69:0/2515635839 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7eff38002000 con 0x7eff4014be90 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.977+0000 7eff3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.977+0000 7eff3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.977+0000 7eff3dffb640 10 monclient: dump: 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:54.977 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:54.978 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:54.978 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:54.978 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:54.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.977+0000 7eff45979640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:54.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.977+0000 7eff45979640 4 set_mon_vals no callback set 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.978+0000 7eff45979640 10 monclient: shutdown 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.978+0000 7eff45979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.978+0000 7eff45979640 1 -- 172.21.15.69:0/2515635839 >> v1:172.21.15.69:6789/0 conn(0x7eff4014be90 legacy=0x7eff4014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.978+0000 7eff45979640 1 -- 172.21.15.69:0/2515635839 shutdown_connections 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.978+0000 7eff45979640 1 -- 172.21.15.69:0/2515635839 >> 172.21.15.69:0/2515635839 conn(0x7eff400617a0 msgr2=0x7eff40061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.978+0000 7eff45979640 1 -- 172.21.15.69:0/2515635839 shutdown_connections 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.978+0000 7eff45979640 1 -- 172.21.15.69:0/2515635839 wait complete. 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.979+0000 7eff45979640 10 monclient: build_initial_monmap 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.979+0000 7eff45979640 10 monclient: monmap: 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.979+0000 7eff45979640 1 librados: starting msgr at 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.979+0000 7eff45979640 1 librados: starting objecter 2024-03-25T19:49:54.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.979+0000 7eff45979640 1 Processor -- start 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 1 -- start start 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 1 librados: setting wanted keys 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 1 librados: calling monclient init 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 10 monclient: init 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 10 monclient(hunting): picked mon.noname-b con 0x7eff400b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 10 monclient(hunting): picked mon.noname-a con 0x7eff400b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 10 monclient(hunting): picked mon.noname-c con 0x7eff4014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff40134a00 con 0x7eff400b4d60 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff40139050 con 0x7eff400b1730 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4013a330 con 0x7eff4014be90 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:54.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:54.980+0000 7eff45979640 10 monclient(hunting): authenticate will time out at 5407.249568s 2024-03-25T19:49:55.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.004+0000 7eff3e7fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7eff400b4d60 0x7eff40133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45262/0 (socket says 172.21.15.69:45262) 2024-03-25T19:49:55.004 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.004+0000 7eff3e7fc640 1 -- 172.21.15.69:0/2880439555 learned_addr learned my addr 172.21.15.69:0/2880439555 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:49:55.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.006+0000 7eff27fff640 10 client.?.objecter ms_handle_connect 0x7eff400b4d60 2024-03-25T19:49:55.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.006+0000 7eff27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:55.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.008+0000 7eff27fff640 10 client.?.objecter ms_handle_connect 0x7eff400b1730 2024-03-25T19:49:55.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.008+0000 7eff27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:55.020 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979731278 0 0) 0x7eff40139050 con 0x7eff400b1730 2024-03-25T19:49:55.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:55.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:55.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:55.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:49:55.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff2c004eb0 con 0x7eff400b1730 2024-03-25T19:49:55.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377296882 0 0) 0x7eff40134a00 con 0x7eff400b4d60 2024-03-25T19:49:55.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:55.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:55.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:55.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:49:55.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.020+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff40139050 con 0x7eff400b4d60 2024-03-25T19:49:55.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.036+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304025431 0 0) 0x7eff2c004eb0 con 0x7eff400b1730 2024-03-25T19:49:55.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.036+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff40134a00 con 0x7eff400b1730 2024-03-25T19:49:55.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.036+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7eff300018f0 con 0x7eff400b1730 2024-03-25T19:49:55.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.036+0000 7eff27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:55.036 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.036+0000 7eff27fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.036+0000 7eff27fff640 10 monclient(hunting): dump: 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:55.037 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.036+0000 7eff27fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:55.038 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.037+0000 7eff27fff640 10 client.?.objecter ms_handle_connect 0x7eff4014be90 2024-03-25T19:49:55.038 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.037+0000 7eff27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:55.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.043+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303922341 0 0) 0x7eff40139050 con 0x7eff400b4d60 2024-03-25T19:49:55.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.043+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff2c004eb0 con 0x7eff400b4d60 2024-03-25T19:49:55.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.043+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7eff34001930 con 0x7eff400b4d60 2024-03-25T19:49:55.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.043+0000 7eff27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:55.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.043+0000 7eff27fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:55.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.043+0000 7eff27fff640 10 monclient(hunting): dump: 2024-03-25T19:49:55.043 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:55.043 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:55.044 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:55.044 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:55.044 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:55.044 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:55.044 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:55.044 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:55.044 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:55.044 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:55.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.045+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131238587 0 0) 0x7eff4013a330 con 0x7eff4014be90 2024-03-25T19:49:55.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.045+0000 7eff27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:55.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.045+0000 7eff27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:55.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.045+0000 7eff27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:55.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.045+0000 7eff27fff640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:49:55.045 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.045+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff40139050 con 0x7eff4014be90 2024-03-25T19:49:55.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 806009095 0 0) 0x7eff40134a00 con 0x7eff400b1730 2024-03-25T19:49:55.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 >> v1:172.21.15.69:6790/0 conn(0x7eff4014be90 legacy=0x7eff401342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 >> v1:172.21.15.69:6789/0 conn(0x7eff400b4d60 legacy=0x7eff40133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:55.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 1 monclient: found mon.b 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4013a610 con 0x7eff400b1730 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff45979640 5 monclient: authenticate success, global_id 4615 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff45979640 10 monclient: _renew_subs 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.049+0000 7eff45979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.050+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4013a8f0 con 0x7eff400b1730 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.050+0000 7eff45979640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.050+0000 7eff45979640 10 monclient: _renew_subs 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.050+0000 7eff45979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.050+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff40143140 con 0x7eff400b1730 2024-03-25T19:49:55.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.050+0000 7eff45979640 1 librados: init done 2024-03-25T19:49:55.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.051+0000 7eff45979640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:55.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.051+0000 7eff45979640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:55.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.051+0000 7eff45979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:55.052 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.051+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff400627b0 con 0x7eff400b1730 2024-03-25T19:49:55.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.333+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff30003610 con 0x7eff400b1730 2024-03-25T19:49:55.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.333+0000 7eff27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:55.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.333+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7eff30003920 con 0x7eff400b1730 2024-03-25T19:49:55.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.333+0000 7eff27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:55.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.333+0000 7eff27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.333+0000 7eff27fff640 10 monclient: dump: 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:49:55.335 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.333+0000 7eff3d7fa640 4 set_mon_vals no callback set 2024-03-25T19:49:55.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.334+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7eff300198f0 con 0x7eff400b1730 2024-03-25T19:49:55.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.335+0000 7eff3e7fc640 10 monclient: get_auth_request con 0x7eff2c06df00 auth_method 0 2024-03-25T19:49:55.336 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.335+0000 7eff27fff640 10 client.4615.objecter ms_handle_connect 0x7eff2c06df00 2024-03-25T19:49:55.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.342+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7eff300535f0 con 0x7eff400b1730 2024-03-25T19:49:55.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.342+0000 7eff27fff640 10 client.4615.objecter ms_dispatch 0x7eff40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:55.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.342+0000 7eff27fff640 3 client.4615.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:55.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.342+0000 7eff27fff640 3 client.4615.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:55.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.342+0000 7eff27fff640 20 client.4615.objecter dump_active .. 0 homeless 2024-03-25T19:49:55.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.343+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff3004ef30 con 0x7eff400b1730 2024-03-25T19:49:55.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.343+0000 7eff27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:55.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.343+0000 7eff27fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:55.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.591+0000 7eff45979640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.591 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.591+0000 7eff45979640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.592 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.591+0000 7eff45979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:49:55.592 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.591+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff401391e0 con 0x7eff400b1730 2024-03-25T19:49:55.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.620+0000 7eff27fff640 1 -- 172.21.15.69:0/2880439555 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7eff401391e0 con 0x7eff400b1730 2024-03-25T19:49:55.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.620+0000 7eff27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.620+0000 7eff27fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:55.620 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:49:55.621 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:55.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 10 librados: watch_flush enter 2024-03-25T19:49:55.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 10 librados: watch_flush exit 2024-03-25T19:49:55.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 20 client.4615.objecter shutdown clearing up homeless session... 2024-03-25T19:49:55.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 10 client.4615.objecter successfully canceled tick 2024-03-25T19:49:55.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 >> v1:172.21.15.169:6800/3072548259 conn(0x7eff2c06df00 legacy=0x7eff2c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 10 monclient: shutdown 2024-03-25T19:49:55.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:55.623 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 >> v1:172.21.15.169:6789/0 conn(0x7eff400b1730 legacy=0x7eff4013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.624 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 shutdown_connections 2024-03-25T19:49:55.624 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 >> 172.21.15.69:0/2880439555 conn(0x7eff400617a0 msgr2=0x7eff400b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:55.624 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 shutdown_connections 2024-03-25T19:49:55.624 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 1 -- 172.21.15.69:0/2880439555 wait complete. 2024-03-25T19:49:55.624 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:49:55.622+0000 7eff45979640 1 librados: shutdown 2024-03-25T19:49:55.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:02.442 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:02.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.555+0000 7f53f68f4640 10 monclient: get_monmap_and_config 2024-03-25T19:50:02.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.555+0000 7f53f68f4640 10 monclient: build_initial_monmap 2024-03-25T19:50:02.554 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.555+0000 7f53f68f4640 10 monclient: monmap: 2024-03-25T19:50:02.554 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:02.554 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:02.554 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:50:02.555265+0000 2024-03-25T19:50:02.554 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:50:02.555265+0000 2024-03-25T19:50:02.554 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:02.555 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:02.555 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:02.555 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:50:02.555 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:50:02.555 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.555+0000 7f53f68f4640 1 Processor -- start 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.555+0000 7f53f68f4640 1 -- start start 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.555+0000 7f53f68f4640 10 monclient: init 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 10 monclient(hunting): picked mon.noname-a con 0x7f53f00a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 10 monclient(hunting): picked mon.noname-b con 0x7f53f013f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 10 monclient(hunting): picked mon.noname-c con 0x7f53f01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:02.556 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f0142520 con 0x7f53f00a98f0 2024-03-25T19:50:02.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00b28a0 con 0x7f53f01430c0 2024-03-25T19:50:02.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00b2a30 con 0x7f53f013f710 2024-03-25T19:50:02.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:02.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.556+0000 7f53f68f4640 10 monclient(hunting): authenticate will time out at 5414.824376s 2024-03-25T19:50:02.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.564+0000 7f53f4e91640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f53f01430c0 0x7f53f014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:51014/0 (socket says 172.21.15.69:51014) 2024-03-25T19:50:02.563 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.564+0000 7f53f4e91640 1 -- 172.21.15.69:0/2309181871 learned_addr learned my addr 172.21.15.69:0/2309181871 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:02.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.577+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236099132 0 0) 0x7f53f00b28a0 con 0x7f53f01430c0 2024-03-25T19:50:02.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.577+0000 7f53eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.577+0000 7f53eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.577+0000 7f53eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.577+0000 7f53eeffd640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:50:02.577 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.577+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d8003520 con 0x7f53f01430c0 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.590+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023346230 0 0) 0x7f53f00b2a30 con 0x7f53f013f710 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.590+0000 7f53eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.590+0000 7f53eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.590+0000 7f53eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.590+0000 7f53eeffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.590+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f00b28a0 con 0x7f53f013f710 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.591+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269321698 0 0) 0x7f53f0142520 con 0x7f53f00a98f0 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.591+0000 7f53eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.591+0000 7f53eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.591+0000 7f53eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.591+0000 7f53eeffd640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:50:02.590 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.591+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f00b2a30 con 0x7f53f00a98f0 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1756901718 0 0) 0x7f53f00b2a30 con 0x7f53f00a98f0 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 >> v1:172.21.15.69:6790/0 conn(0x7f53f013f710 legacy=0x7f53f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 >> v1:172.21.15.169:6789/0 conn(0x7f53f01430c0 legacy=0x7f53f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 1 monclient: found mon.noname-a 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00b44c0 con 0x7f53f00a98f0 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:02.606 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f53e00015b0 con 0x7f53f00a98f0 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53eeffd640 10 monclient: dump: 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53f68f4640 5 monclient: authenticate success, global_id 4650 2024-03-25T19:50:02.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.606+0000 7f53f68f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:02.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.640+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53e0001710 con 0x7f53f00a98f0 2024-03-25T19:50:02.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.640+0000 7f53eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:02.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.640+0000 7f53eeffd640 1 -- 172.21.15.69:0/2309181871 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f53e0001b60 con 0x7f53f00a98f0 2024-03-25T19:50:02.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.640+0000 7f53eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.640+0000 7f53eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:02.639 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.640+0000 7f53eeffd640 10 monclient: dump: 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.640+0000 7f53f68f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:02.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.640+0000 7f53f68f4640 4 set_mon_vals no callback set 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.641+0000 7f53f68f4640 10 monclient: shutdown 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.641+0000 7f53f68f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.641+0000 7f53f68f4640 1 -- 172.21.15.69:0/2309181871 >> v1:172.21.15.69:6789/0 conn(0x7f53f00a98f0 legacy=0x7f53f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.641+0000 7f53f68f4640 1 -- 172.21.15.69:0/2309181871 shutdown_connections 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.641+0000 7f53f68f4640 1 -- 172.21.15.69:0/2309181871 >> 172.21.15.69:0/2309181871 conn(0x7f53f00b7ea0 msgr2=0x7f53f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.641+0000 7f53f68f4640 1 -- 172.21.15.69:0/2309181871 shutdown_connections 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.641+0000 7f53f68f4640 1 -- 172.21.15.69:0/2309181871 wait complete. 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 10 monclient: build_initial_monmap 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 10 monclient: monmap: 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:02.642 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 1 librados: starting msgr at 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 1 librados: starting objecter 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 1 Processor -- start 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 1 -- start start 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 1 librados: setting wanted keys 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 1 librados: calling monclient init 2024-03-25T19:50:02.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.642+0000 7f53f68f4640 10 monclient: init 2024-03-25T19:50:02.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:02.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:02.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 10 monclient(hunting): picked mon.noname-c con 0x7f53f00a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:02.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 10 monclient(hunting): picked mon.noname-a con 0x7f53f013f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:02.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 10 monclient(hunting): picked mon.noname-b con 0x7f53f01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:02.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f01dab90 con 0x7f53f013f710 2024-03-25T19:50:02.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00abac0 con 0x7f53f01430c0 2024-03-25T19:50:02.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f01db730 con 0x7f53f00a98f0 2024-03-25T19:50:02.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:02.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.643+0000 7f53f68f4640 10 monclient(hunting): authenticate will time out at 5414.911265s 2024-03-25T19:50:02.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.649+0000 7f53effff640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f53f00a98f0 0x7f53f01db020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:58056/0 (socket says 172.21.15.69:58056) 2024-03-25T19:50:02.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.649+0000 7f53effff640 1 -- 172.21.15.69:0/665300794 learned_addr learned my addr 172.21.15.69:0/665300794 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:02.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.652+0000 7f53ecff9640 10 client.?.objecter ms_handle_connect 0x7f53f00a98f0 2024-03-25T19:50:02.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.652+0000 7f53ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.657+0000 7f53ecff9640 10 client.?.objecter ms_handle_connect 0x7f53f01430c0 2024-03-25T19:50:02.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.657+0000 7f53ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.661+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885745312 0 0) 0x7f53f00abac0 con 0x7f53f01430c0 2024-03-25T19:50:02.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.661+0000 7f53ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.661+0000 7f53ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.661+0000 7f53ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.661+0000 7f53ecff9640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:50:02.661 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.661+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53dc004eb0 con 0x7f53f01430c0 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.662+0000 7f53ecff9640 10 client.?.objecter ms_handle_connect 0x7f53f013f710 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.662+0000 7f53ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.663+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290315730 0 0) 0x7f53f01db730 con 0x7f53f00a98f0 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.663+0000 7f53ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.663+0000 7f53ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.663+0000 7f53ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.663+0000 7f53ecff9640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.663+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f00abac0 con 0x7f53f00a98f0 2024-03-25T19:50:02.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.675+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288092517 0 0) 0x7f53f01dab90 con 0x7f53f013f710 2024-03-25T19:50:02.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.675+0000 7f53ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.675+0000 7f53ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.675+0000 7f53ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.674 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.675+0000 7f53ecff9640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:50:02.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.675+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f01db730 con 0x7f53f013f710 2024-03-25T19:50:02.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2350350134 0 0) 0x7f53f00abac0 con 0x7f53f00a98f0 2024-03-25T19:50:02.677 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53f01dab90 con 0x7f53f00a98f0 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f53e0001a60 con 0x7f53f00a98f0 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 10 monclient(hunting): dump: 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:02.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2217890954 0 0) 0x7f53dc004eb0 con 0x7f53f01430c0 2024-03-25T19:50:02.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.678+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53f00abac0 con 0x7f53f01430c0 2024-03-25T19:50:02.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.679+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f53e8001b90 con 0x7f53f01430c0 2024-03-25T19:50:02.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.679+0000 7f53ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.679+0000 7f53ecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:02.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.679+0000 7f53ecff9640 10 monclient(hunting): dump: 2024-03-25T19:50:02.679 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:02.679 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:02.680 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.680 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.680 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.680 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:02.680 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:02.680 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:02.680 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:02.680 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2255482741 0 0) 0x7f53f00abac0 con 0x7f53f01430c0 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 >> v1:172.21.15.69:6790/0 conn(0x7f53f00a98f0 legacy=0x7f53f01db020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 >> v1:172.21.15.69:6789/0 conn(0x7f53f013f710 legacy=0x7f53f0138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 1 monclient: found mon.b 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00abd30 con 0x7f53f01430c0 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:02.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53f68f4640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53f68f4640 10 monclient: _renew_subs 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53f68f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.688+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f0138c90 con 0x7f53f01430c0 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 10 monclient: _renew_subs 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f0137bb0 con 0x7f53f01430c0 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 1 librados: init done 2024-03-25T19:50:02.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:02.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.689+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f00b3ef0 con 0x7f53f01430c0 2024-03-25T19:50:02.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.975+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53e8002060 con 0x7f53f01430c0 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.975+0000 7f53ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.975+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f53e8004d00 con 0x7f53f01430c0 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.975+0000 7f53ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.975+0000 7f53ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.975+0000 7f53ecff9640 10 monclient: dump: 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:02.976 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:02.977 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:02.977 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:02.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.975+0000 7f53ee7fc640 4 set_mon_vals no callback set 2024-03-25T19:50:02.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.976+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f53e801ad10 con 0x7f53f01430c0 2024-03-25T19:50:02.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.977+0000 7f53effff640 10 monclient: get_auth_request con 0x7f53dc06df40 auth_method 0 2024-03-25T19:50:02.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.977+0000 7f53ecff9640 10 client.4621.objecter ms_handle_connect 0x7f53dc06df40 2024-03-25T19:50:02.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.982+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f53e8054ab0 con 0x7f53f01430c0 2024-03-25T19:50:02.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.982+0000 7f53ecff9640 10 client.4621.objecter ms_dispatch 0x7f53f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:02.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.982+0000 7f53ecff9640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:02.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.982+0000 7f53ecff9640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:02.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.982+0000 7f53ecff9640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T19:50:02.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.982+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53e80503f0 con 0x7f53f01430c0 2024-03-25T19:50:02.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.982+0000 7f53ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:02.982+0000 7f53ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:03.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.196+0000 7f53f68f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.196+0000 7f53f68f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.196+0000 7f53f68f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:03.195 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.196+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53f01da8f0 con 0x7f53f01430c0 2024-03-25T19:50:03.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.263+0000 7f53ecff9640 1 -- 172.21.15.69:0/665300794 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f53e80524f0 con 0x7f53f01430c0 2024-03-25T19:50:03.262 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.263+0000 7f53ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.263+0000 7f53ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:03.263 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:50:03.263 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 10 librados: watch_flush enter 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 10 librados: watch_flush exit 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 10 client.4621.objecter successfully canceled tick 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 >> v1:172.21.15.169:6800/3072548259 conn(0x7f53dc06df40 legacy=0x7f53dc070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 10 monclient: shutdown 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 >> v1:172.21.15.169:6789/0 conn(0x7f53f01430c0 legacy=0x7f53f01e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 shutdown_connections 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 >> 172.21.15.69:0/665300794 conn(0x7f53f00b7ea0 msgr2=0x7f53f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 shutdown_connections 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.266+0000 7f53f68f4640 1 -- 172.21.15.69:0/665300794 wait complete. 2024-03-25T19:50:03.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:03.267+0000 7f53f68f4640 1 librados: shutdown 2024-03-25T19:50:03.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:10.086 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:10.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.200+0000 7fbb77606640 10 monclient: get_monmap_and_config 2024-03-25T19:50:10.200 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.200+0000 7fbb77606640 10 monclient: build_initial_monmap 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.200+0000 7fbb77606640 10 monclient: monmap: 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:50:10.201475+0000 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:50:10.201475+0000 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:50:10.201 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.201+0000 7fbb77606640 1 Processor -- start 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.201+0000 7fbb77606640 1 -- start start 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.201+0000 7fbb77606640 10 monclient: init 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 10 monclient(hunting): picked mon.noname-a con 0x7fbb700a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 10 monclient(hunting): picked mon.noname-c con 0x7fbb7013f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 10 monclient(hunting): picked mon.noname-b con 0x7fbb701430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb70142520 con 0x7fbb700a98f0 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb700b28a0 con 0x7fbb7013f710 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb700b2a30 con 0x7fbb701430c0 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:10.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.202+0000 7fbb77606640 10 monclient(hunting): authenticate will time out at 5422.471034s 2024-03-25T19:50:10.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.212+0000 7fbb75ba3640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fbb701430c0 0x7fbb7014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:34572/0 (socket says 172.21.15.69:34572) 2024-03-25T19:50:10.212 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.212+0000 7fbb75ba3640 1 -- 172.21.15.69:0/695099955 learned_addr learned my addr 172.21.15.69:0/695099955 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:10.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.217+0000 7fbb75ba3640 1 -- 172.21.15.69:0/695099955 >> v1:172.21.15.69:6790/0 conn(0x7fbb701430c0 legacy=0x7fbb7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:50:10.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.217+0000 7fbb75ba3640 1 -- 172.21.15.69:0/695099955 >> v1:172.21.15.69:6790/0 conn(0x7fbb701430c0 legacy=0x7fbb7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:50:10.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.217+0000 7fbb75ba3640 1 --1- 172.21.15.69:0/695099955 >> v1:172.21.15.69:6790/0 conn(0x7fbb701430c0 0x7fbb7014dc30 :-1 s=OPENED pgs=70107 cs=1 l=1).handle_message read tag failed 2024-03-25T19:50:10.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.217+0000 7fbb75ba3640 1 --1- 172.21.15.69:0/695099955 >> v1:172.21.15.69:6790/0 conn(0x7fbb701430c0 0x7fbb7014dc30 :-1 s=OPENED pgs=70107 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:50:10.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.217+0000 7fbb67fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.69:6790/0 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.229+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717291957 0 0) 0x7fbb70142520 con 0x7fbb700a98f0 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.229+0000 7fbb67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.229+0000 7fbb67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.229+0000 7fbb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.229+0000 7fbb67fff640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.229+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c003520 con 0x7fbb700a98f0 2024-03-25T19:50:10.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.241+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134453030 0 0) 0x7fbb700b28a0 con 0x7fbb7013f710 2024-03-25T19:50:10.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.241+0000 7fbb67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.241+0000 7fbb67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.241+0000 7fbb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.241+0000 7fbb67fff640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:50:10.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.241+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb70142520 con 0x7fbb7013f710 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4091342175 0 0) 0x7fbb5c003520 con 0x7fbb700a98f0 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 >> v1:172.21.15.69:6790/0 conn(0x7fbb701430c0 legacy=0x7fbb7014dc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 >> v1:172.21.15.169:6789/0 conn(0x7fbb7013f710 legacy=0x7fbb701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 1 monclient: found mon.noname-a 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb700b44c0 con 0x7fbb700a98f0 2024-03-25T19:50:10.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 10 monclient: _finish_auth 0 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fbb6c001690 con 0x7fbb700a98f0 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb77606640 5 monclient: authenticate success, global_id 4656 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb77606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb67fff640 10 monclient: dump: 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:10.267 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:10.268 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.268 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:10.268 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:10.268 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:10.268 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:10.268 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:10.268 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.265+0000 7fbb77606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:10.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.291+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb6c0017f0 con 0x7fbb700a98f0 2024-03-25T19:50:10.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.291+0000 7fbb67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:10.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.291+0000 7fbb67fff640 1 -- 172.21.15.69:0/695099955 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fbb6c001c00 con 0x7fbb700a98f0 2024-03-25T19:50:10.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.291+0000 7fbb67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.291+0000 7fbb67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:10.291 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.291+0000 7fbb67fff640 10 monclient: dump: 2024-03-25T19:50:10.291 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:10.291 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.291+0000 7fbb77606640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:10.292 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.291+0000 7fbb77606640 4 set_mon_vals no callback set 2024-03-25T19:50:10.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.292+0000 7fbb77606640 10 monclient: shutdown 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.292+0000 7fbb77606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.292+0000 7fbb77606640 1 -- 172.21.15.69:0/695099955 >> v1:172.21.15.69:6789/0 conn(0x7fbb700a98f0 legacy=0x7fbb700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.292+0000 7fbb77606640 1 -- 172.21.15.69:0/695099955 shutdown_connections 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.292+0000 7fbb77606640 1 -- 172.21.15.69:0/695099955 >> 172.21.15.69:0/695099955 conn(0x7fbb700b7ea0 msgr2=0x7fbb700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.292+0000 7fbb77606640 1 -- 172.21.15.69:0/695099955 shutdown_connections 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.292+0000 7fbb77606640 1 -- 172.21.15.69:0/695099955 wait complete. 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.292+0000 7fbb77606640 10 monclient: build_initial_monmap 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.293+0000 7fbb77606640 10 monclient: monmap: 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.293+0000 7fbb77606640 1 librados: starting msgr at 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.293+0000 7fbb77606640 1 librados: starting objecter 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.293+0000 7fbb77606640 1 Processor -- start 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.293+0000 7fbb77606640 1 -- start start 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.293+0000 7fbb77606640 1 librados: setting wanted keys 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.293+0000 7fbb77606640 1 librados: calling monclient init 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.293+0000 7fbb77606640 10 monclient: init 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 10 monclient(hunting): picked mon.noname-a con 0x7fbb700a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 10 monclient(hunting): picked mon.noname-b con 0x7fbb7013f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 10 monclient(hunting): picked mon.noname-c con 0x7fbb701430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb701d7d20 con 0x7fbb700a98f0 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb700ad240 con 0x7fbb7013f710 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb701d48f0 con 0x7fbb701430c0 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.294+0000 7fbb77606640 10 monclient(hunting): authenticate will time out at 5422.562871s 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.296+0000 7fbb75ba3640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fbb701430c0 0x7fbb701d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:34626/0 (socket says 172.21.15.69:34626) 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.296+0000 7fbb75ba3640 1 -- 172.21.15.69:0/1771027737 learned_addr learned my addr 172.21.15.69:0/1771027737 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:10.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.298+0000 7fbb65ffb640 10 client.?.objecter ms_handle_connect 0x7fbb701430c0 2024-03-25T19:50:10.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.298+0000 7fbb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.305+0000 7fbb65ffb640 10 client.?.objecter ms_handle_connect 0x7fbb700a98f0 2024-03-25T19:50:10.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.305+0000 7fbb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.306+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064594861 0 0) 0x7fbb701d48f0 con 0x7fbb701430c0 2024-03-25T19:50:10.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.306+0000 7fbb65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.306+0000 7fbb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.306+0000 7fbb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.306+0000 7fbb65ffb640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:50:10.306 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.306+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb54004eb0 con 0x7fbb701430c0 2024-03-25T19:50:10.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.313+0000 7fbb65ffb640 10 client.?.objecter ms_handle_connect 0x7fbb7013f710 2024-03-25T19:50:10.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.313+0000 7fbb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.316+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184638599 0 0) 0x7fbb54004eb0 con 0x7fbb701430c0 2024-03-25T19:50:10.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.316+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb701d48f0 con 0x7fbb701430c0 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.316+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fbb68001b90 con 0x7fbb701430c0 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.316+0000 7fbb65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.316+0000 7fbb65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.316+0000 7fbb65ffb640 10 monclient(hunting): dump: 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:10.317 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.316+0000 7fbb65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:10.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.321+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143145775 0 0) 0x7fbb701d7d20 con 0x7fbb700a98f0 2024-03-25T19:50:10.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.321+0000 7fbb65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.321+0000 7fbb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.321+0000 7fbb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.321+0000 7fbb65ffb640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:50:10.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.321+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb54004eb0 con 0x7fbb700a98f0 2024-03-25T19:50:10.329 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.329+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153725089 0 0) 0x7fbb700ad240 con 0x7fbb7013f710 2024-03-25T19:50:10.329 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.329+0000 7fbb65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.329+0000 7fbb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.329+0000 7fbb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.329+0000 7fbb65ffb640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:50:10.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.329+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb701d7d20 con 0x7fbb7013f710 2024-03-25T19:50:10.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.337+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1433970188 0 0) 0x7fbb701d48f0 con 0x7fbb701430c0 2024-03-25T19:50:10.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 >> v1:172.21.15.169:6789/0 conn(0x7fbb7013f710 legacy=0x7fbb700acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 >> v1:172.21.15.69:6789/0 conn(0x7fbb700a98f0 legacy=0x7fbb700b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 1 monclient: found mon.c 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb701d5180 con 0x7fbb701430c0 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 5 monclient: authenticate success, global_id 4628 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 10 monclient: _renew_subs 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:10.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb701d81b0 con 0x7fbb701430c0 2024-03-25T19:50:10.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:10.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 10 monclient: _renew_subs 2024-03-25T19:50:10.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:10.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7014e120 con 0x7fbb701430c0 2024-03-25T19:50:10.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.338+0000 7fbb77606640 1 librados: init done 2024-03-25T19:50:10.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.339+0000 7fbb77606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.339+0000 7fbb77606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.339+0000 7fbb77606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:10.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.339+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb701d8550 con 0x7fbb701430c0 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.643+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb680020c0 con 0x7fbb701430c0 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.643+0000 7fbb65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.643+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fbb68004b60 con 0x7fbb701430c0 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.643+0000 7fbb65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.643+0000 7fbb65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.643+0000 7fbb65ffb640 10 monclient: dump: 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:10.644 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:10.645 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:10.645 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.645 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:10.645 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:10.645 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:10.645 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:10.645 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:10.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.643+0000 7fbb677fe640 4 set_mon_vals no callback set 2024-03-25T19:50:10.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.644+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fbb68004d80 con 0x7fbb701430c0 2024-03-25T19:50:10.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.645+0000 7fbb753a2640 10 monclient: get_auth_request con 0x7fbb5406de10 auth_method 0 2024-03-25T19:50:10.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.645+0000 7fbb65ffb640 10 client.4628.objecter ms_handle_connect 0x7fbb5406de10 2024-03-25T19:50:10.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.649+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fbb680549c0 con 0x7fbb701430c0 2024-03-25T19:50:10.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.649+0000 7fbb65ffb640 10 client.4628.objecter ms_dispatch 0x7fbb70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:10.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.649+0000 7fbb65ffb640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:10.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.649+0000 7fbb65ffb640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:10.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.649+0000 7fbb65ffb640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T19:50:10.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.649+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb68050300 con 0x7fbb701430c0 2024-03-25T19:50:10.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.649+0000 7fbb65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.649+0000 7fbb65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:10.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.867+0000 7fbb77606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.867+0000 7fbb77606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.867+0000 7fbb77606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:10.868 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:10.867+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb700b16b0 con 0x7fbb701430c0 2024-03-25T19:50:11.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.029+0000 7fbb65ffb640 1 -- 172.21.15.69:0/1771027737 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fbb68052400 con 0x7fbb701430c0 2024-03-25T19:50:11.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.029+0000 7fbb65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:11.029 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.029+0000 7fbb65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:11.029 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:50:11.030 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:11.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 10 librados: watch_flush enter 2024-03-25T19:50:11.031 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 10 librados: watch_flush exit 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 10 client.4628.objecter successfully canceled tick 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 >> v1:172.21.15.169:6800/3072548259 conn(0x7fbb5406de10 legacy=0x7fbb540702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 10 monclient: shutdown 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 >> v1:172.21.15.69:6790/0 conn(0x7fbb701430c0 legacy=0x7fbb701d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 shutdown_connections 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 >> 172.21.15.69:0/1771027737 conn(0x7fbb700b7ea0 msgr2=0x7fbb70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 shutdown_connections 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 1 -- 172.21.15.69:0/1771027737 wait complete. 2024-03-25T19:50:11.032 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:11.031+0000 7fbb77606640 1 librados: shutdown 2024-03-25T19:50:11.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:13.678 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:51.642 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.686 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:51.650 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.721 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:51.685 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.737 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:51.701 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.745 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:51.699 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.746 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:51.710 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.750 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:51.704 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.769 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:51.733 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.784 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:51.738 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:17.849 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.961+0000 7f6cbe354640 10 monclient: get_monmap_and_config 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.961+0000 7f6cbe354640 10 monclient: build_initial_monmap 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.961+0000 7f6cbe354640 10 monclient: monmap: 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:50:17.962488+0000 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:50:17.962488+0000 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:17.961 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:17.962 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:50:17.962 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:50:17.962 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 1 Processor -- start 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 1 -- start start 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 10 monclient: init 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb80b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb80b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb814be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80af940 con 0x7f6cb80b1730 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80b0a70 con 0x7f6cb80b4d60 2024-03-25T19:50:17.963 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80b0d50 con 0x7f6cb814be90 2024-03-25T19:50:17.964 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:17.964 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.962+0000 7f6cbe354640 10 monclient(hunting): authenticate will time out at 5430.231509s 2024-03-25T19:50:17.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.975+0000 7f6cbc8f1640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f6cb814be90 0x7f6cb814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:46806/0 (socket says 172.21.15.69:46806) 2024-03-25T19:50:17.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.975+0000 7f6cbc8f1640 1 -- 172.21.15.69:0/3630415121 learned_addr learned my addr 172.21.15.69:0/3630415121 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:17.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.982+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989151227 0 0) 0x7f6cb80af940 con 0x7f6cb80b1730 2024-03-25T19:50:17.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.982+0000 7f6cb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.982+0000 7f6cb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.982+0000 7f6cb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.982+0000 7f6cb6ffd640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:50:17.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.982+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca0003520 con 0x7f6cb80b1730 2024-03-25T19:50:17.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.992+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223149458 0 0) 0x7f6cb80b0a70 con 0x7f6cb80b4d60 2024-03-25T19:50:17.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.992+0000 7f6cb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.992+0000 7f6cb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.992+0000 7f6cb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.992+0000 7f6cb6ffd640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:50:17.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.992+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb80af940 con 0x7f6cb80b4d60 2024-03-25T19:50:17.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.996+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930205854 0 0) 0x7f6cb80b0d50 con 0x7f6cb814be90 2024-03-25T19:50:17.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.996+0000 7f6cb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.996+0000 7f6cb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.996+0000 7f6cb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.996+0000 7f6cb6ffd640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:50:17.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.996+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb80b0a70 con 0x7f6cb814be90 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2350145254 0 0) 0x7f6ca0003520 con 0x7f6cb80b1730 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 >> v1:172.21.15.69:6790/0 conn(0x7f6cb814be90 legacy=0x7f6cb814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 >> v1:172.21.15.169:6789/0 conn(0x7f6cb80b4d60 legacy=0x7f6cb813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb80b1030 con 0x7f6cb80b1730 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:17.999 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6cac001690 con 0x7f6cb80b1730 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cbe354640 5 monclient: authenticate success, global_id 4662 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cb6ffd640 10 monclient: dump: 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:18.000 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:18.001 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:18.001 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:17.999+0000 7f6cbe354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:18.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.018+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cac001870 con 0x7f6cb80b1730 2024-03-25T19:50:18.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.018+0000 7f6cb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:18.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.018+0000 7f6cb6ffd640 1 -- 172.21.15.69:0/3630415121 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6cac001c80 con 0x7f6cb80b1730 2024-03-25T19:50:18.018 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.018+0000 7f6cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.018+0000 7f6cb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.018+0000 7f6cb6ffd640 10 monclient: dump: 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.018+0000 7f6cbe354640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:18.019 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.018+0000 7f6cbe354640 4 set_mon_vals no callback set 2024-03-25T19:50:18.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.019+0000 7f6cbe354640 10 monclient: shutdown 2024-03-25T19:50:18.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.019+0000 7f6cbe354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:18.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.019+0000 7f6cbe354640 1 -- 172.21.15.69:0/3630415121 >> v1:172.21.15.69:6789/0 conn(0x7f6cb80b1730 legacy=0x7f6cb80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.019+0000 7f6cbe354640 1 -- 172.21.15.69:0/3630415121 shutdown_connections 2024-03-25T19:50:18.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.019+0000 7f6cbe354640 1 -- 172.21.15.69:0/3630415121 >> 172.21.15.69:0/3630415121 conn(0x7f6cb80617a0 msgr2=0x7f6cb8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.019+0000 7f6cbe354640 1 -- 172.21.15.69:0/3630415121 shutdown_connections 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.019+0000 7f6cbe354640 1 -- 172.21.15.69:0/3630415121 wait complete. 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 10 monclient: build_initial_monmap 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 10 monclient: monmap: 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:50:18.022 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 1 librados: starting msgr at 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 1 librados: starting objecter 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 1 Processor -- start 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 1 -- start start 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 1 librados: setting wanted keys 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 1 librados: calling monclient init 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.020+0000 7f6cbe354640 10 monclient: init 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb80b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb80b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb814be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb81307c0 con 0x7f6cb80b4d60 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb8131c20 con 0x7f6cb814be90 2024-03-25T19:50:18.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb8131db0 con 0x7f6cb80b1730 2024-03-25T19:50:18.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:18.024 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.021+0000 7f6cbe354640 10 monclient(hunting): authenticate will time out at 5430.290190s 2024-03-25T19:50:18.041 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.041+0000 7f6cbc8f1640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f6cb814be90 0x7f6cb81300b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:51168/0 (socket says 172.21.15.69:51168) 2024-03-25T19:50:18.041 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.041+0000 7f6cbc8f1640 1 -- 172.21.15.69:0/604149869 learned_addr learned my addr 172.21.15.69:0/604149869 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:18.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.043+0000 7f6cb4ff9640 10 client.?.objecter ms_handle_connect 0x7f6cb814be90 2024-03-25T19:50:18.043 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.043+0000 7f6cb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:18.046 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.047+0000 7f6cb4ff9640 10 client.?.objecter ms_handle_connect 0x7f6cb80b1730 2024-03-25T19:50:18.047 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.047+0000 7f6cb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:18.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.048+0000 7f6cb4ff9640 10 client.?.objecter ms_handle_connect 0x7f6cb80b4d60 2024-03-25T19:50:18.048 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.048+0000 7f6cb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:18.050 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.050+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155159353 0 0) 0x7f6cb8131c20 con 0x7f6cb814be90 2024-03-25T19:50:18.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.050+0000 7f6cb4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:18.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.050+0000 7f6cb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:18.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.050+0000 7f6cb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:18.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.050+0000 7f6cb4ff9640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:50:18.051 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.050+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca4004eb0 con 0x7f6cb814be90 2024-03-25T19:50:18.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.056+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104348380 0 0) 0x7f6cb81307c0 con 0x7f6cb80b4d60 2024-03-25T19:50:18.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.056+0000 7f6cb4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:18.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.056+0000 7f6cb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:18.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.056+0000 7f6cb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:18.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.056+0000 7f6cb4ff9640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:50:18.056 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.056+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb8131c20 con 0x7f6cb80b4d60 2024-03-25T19:50:18.063 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.063+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3895233775 0 0) 0x7f6ca4004eb0 con 0x7f6cb814be90 2024-03-25T19:50:18.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.064+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb81307c0 con 0x7f6cb814be90 2024-03-25T19:50:18.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.064+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406348749 0 0) 0x7f6cb8131db0 con 0x7f6cb80b1730 2024-03-25T19:50:18.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.064+0000 7f6cb4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:18.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.064+0000 7f6cb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:18.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.064+0000 7f6cb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:18.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.064+0000 7f6cb4ff9640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:50:18.064 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.064+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca4004eb0 con 0x7f6cb80b1730 2024-03-25T19:50:18.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.074+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2563563246 0 0) 0x7f6cb8131c20 con 0x7f6cb80b4d60 2024-03-25T19:50:18.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.074+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb8131db0 con 0x7f6cb80b4d60 2024-03-25T19:50:18.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.074+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6cac001b80 con 0x7f6cb80b4d60 2024-03-25T19:50:18.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.074+0000 7f6cb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:18.074 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.074+0000 7f6cb4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.074+0000 7f6cb4ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:18.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.074+0000 7f6cb4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6cb0001c30 con 0x7f6cb814be90 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:18.081 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2942669204 0 0) 0x7f6cb81307c0 con 0x7f6cb814be90 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 >> v1:172.21.15.69:6790/0 conn(0x7f6cb80b1730 legacy=0x7f6cb81d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 >> v1:172.21.15.69:6789/0 conn(0x7f6cb80b4d60 legacy=0x7f6cb812f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 1 monclient: found mon.b 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb8131f40 con 0x7f6cb814be90 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:18.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 10 monclient: _renew_subs 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb813ed20 con 0x7f6cb814be90 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 10 monclient: _renew_subs 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb8143900 con 0x7f6cb814be90 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.080+0000 7f6cbe354640 1 librados: init done 2024-03-25T19:50:18.083 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.081+0000 7f6cbe354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:18.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.081+0000 7f6cbe354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:18.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.081+0000 7f6cbe354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:18.084 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.081+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb81401f0 con 0x7f6cb814be90 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.372+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cb0004b60 con 0x7f6cb814be90 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.372+0000 7f6cb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.372+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f6cb0002100 con 0x7f6cb814be90 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.372+0000 7f6cb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.372+0000 7f6cb4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.373+0000 7f6cb4ff9640 10 monclient: dump: 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:18.373 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.374 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:18.374 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:18.374 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:18.374 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:18.374 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:18.374 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:18.374 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:18.374 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.372+0000 7f6cb67fc640 4 set_mon_vals no callback set 2024-03-25T19:50:18.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.374+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f6cb0004ff0 con 0x7f6cb814be90 2024-03-25T19:50:18.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.374+0000 7f6cb7fff640 10 monclient: get_auth_request con 0x7f6ca406df70 auth_method 0 2024-03-25T19:50:18.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.375+0000 7f6cb4ff9640 10 client.4633.objecter ms_handle_connect 0x7f6ca406df70 2024-03-25T19:50:18.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.381+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f6cb0054a30 con 0x7f6cb814be90 2024-03-25T19:50:18.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.381+0000 7f6cb4ff9640 10 client.4633.objecter ms_dispatch 0x7f6cb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:18.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.381+0000 7f6cb4ff9640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:18.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.381+0000 7f6cb4ff9640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:18.381 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.381+0000 7f6cb4ff9640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T19:50:18.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.382+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6cb0050540 con 0x7f6cb814be90 2024-03-25T19:50:18.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.382+0000 7f6cb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:18.382 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.382+0000 7f6cb4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:18.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.612+0000 7f6cbe354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.612+0000 7f6cbe354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.612+0000 7f6cbe354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:18.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.612+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cb80627b0 con 0x7f6cb814be90 2024-03-25T19:50:18.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.676+0000 7f6cb4ff9640 1 -- 172.21.15.69:0/604149869 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f6cb0052470 con 0x7f6cb814be90 2024-03-25T19:50:18.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.676+0000 7f6cb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.676+0000 7f6cb4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:18.676 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:50:18.676 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:18.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 10 librados: watch_flush enter 2024-03-25T19:50:18.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 10 librados: watch_flush exit 2024-03-25T19:50:18.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T19:50:18.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 10 client.4633.objecter successfully canceled tick 2024-03-25T19:50:18.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 >> v1:172.21.15.169:6800/3072548259 conn(0x7f6ca406df70 legacy=0x7f6ca4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 10 monclient: shutdown 2024-03-25T19:50:18.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:18.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 >> v1:172.21.15.169:6789/0 conn(0x7f6cb814be90 legacy=0x7f6cb81300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.678+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 shutdown_connections 2024-03-25T19:50:18.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.679+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 >> 172.21.15.69:0/604149869 conn(0x7f6cb80617a0 msgr2=0x7f6cb80b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:18.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.679+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 shutdown_connections 2024-03-25T19:50:18.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.679+0000 7f6cbe354640 1 -- 172.21.15.69:0/604149869 wait complete. 2024-03-25T19:50:18.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:18.679+0000 7f6cbe354640 1 librados: shutdown 2024-03-25T19:50:18.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:25.497 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:25.607 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.607+0000 7f4c3109d640 10 monclient: get_monmap_and_config 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.607+0000 7f4c3109d640 10 monclient: build_initial_monmap 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.607+0000 7f4c3109d640 10 monclient: monmap: 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:50:25.609101+0000 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:50:25.609101+0000 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:50:25.608 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.608+0000 7f4c3109d640 1 Processor -- start 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.608+0000 7f4c3109d640 1 -- start start 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.608+0000 7f4c3109d640 10 monclient: init 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c0b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c0b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c14be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0af940 con 0x7f4c2c14be90 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0b0a70 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0b0d50 con 0x7f4c2c0b1730 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.609+0000 7f4c3109d640 10 monclient(hunting): authenticate will time out at 5437.878087s 2024-03-25T19:50:25.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.612+0000 7f4c2ad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4c2c0b4d60 0x7f4c2c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:60348/0 (socket says 172.21.15.69:60348) 2024-03-25T19:50:25.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.612+0000 7f4c2ad9d640 1 -- 172.21.15.69:0/3882654638 learned_addr learned my addr 172.21.15.69:0/3882654638 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:25.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.618+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573482296 0 0) 0x7f4c2c0b0a70 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.618+0000 7f4c29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.618+0000 7f4c29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.618+0000 7f4c29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.618+0000 7f4c29d9b640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:50:25.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.618+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c14003520 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2935171725 0 0) 0x7f4c14003520 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 >> v1:172.21.15.69:6790/0 conn(0x7f4c2c0b1730 legacy=0x7f4c2c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 >> v1:172.21.15.69:6789/0 conn(0x7f4c2c14be90 legacy=0x7f4c2c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:50:25.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:50:25.632 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 1 monclient: found mon.noname-c 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0b1030 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 10 monclient: _finish_auth 0 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.631+0000 7f4c3109d640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.632+0000 7f4c3109d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.632+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4c1c001690 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.632+0000 7f4c29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.632+0000 7f4c29d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.632+0000 7f4c29d9b640 10 monclient: dump: 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:25.633 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:25.634 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:25.634 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.634 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:25.634 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:25.634 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:25.634 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:25.634 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:25.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.632+0000 7f4c3109d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:25.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.647+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c1c001960 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.647+0000 7f4c29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:25.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.647+0000 7f4c29d9b640 1 -- 172.21.15.69:0/3882654638 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4c1c001cf0 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.647 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.647+0000 7f4c3109d640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.647+0000 7f4c3109d640 4 set_mon_vals no callback set 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.647+0000 7f4c29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.647+0000 7f4c29d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.647+0000 7f4c29d9b640 10 monclient: dump: 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:25.648 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:25.649 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.648+0000 7f4c3109d640 10 monclient: shutdown 2024-03-25T19:50:25.649 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.648+0000 7f4c3109d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:25.649 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.648+0000 7f4c3109d640 1 -- 172.21.15.69:0/3882654638 >> v1:172.21.15.169:6789/0 conn(0x7f4c2c0b4d60 legacy=0x7f4c2c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.649 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.648+0000 7f4c3109d640 1 -- 172.21.15.69:0/3882654638 shutdown_connections 2024-03-25T19:50:25.649 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.648+0000 7f4c3109d640 1 -- 172.21.15.69:0/3882654638 >> 172.21.15.69:0/3882654638 conn(0x7f4c2c0617a0 msgr2=0x7f4c2c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:25.649 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.648+0000 7f4c3109d640 1 -- 172.21.15.69:0/3882654638 shutdown_connections 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.648+0000 7f4c3109d640 1 -- 172.21.15.69:0/3882654638 wait complete. 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.649+0000 7f4c3109d640 10 monclient: build_initial_monmap 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.649+0000 7f4c3109d640 10 monclient: monmap: 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.649+0000 7f4c3109d640 1 librados: starting msgr at 2024-03-25T19:50:25.650 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.649+0000 7f4c3109d640 1 librados: starting objecter 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 1 Processor -- start 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 1 -- start start 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 1 librados: setting wanted keys 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 1 librados: calling monclient init 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 10 monclient: init 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c0b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c0b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c14be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:25.652 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.650+0000 7f4c3109d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c13b910 con 0x7f4c2c14be90 2024-03-25T19:50:25.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.651+0000 7f4c3109d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c1d7940 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.651+0000 7f4c3109d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c13a9d0 con 0x7f4c2c0b1730 2024-03-25T19:50:25.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.651+0000 7f4c3109d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:25.653 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.651+0000 7f4c3109d640 10 monclient(hunting): authenticate will time out at 5437.919647s 2024-03-25T19:50:25.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.655+0000 7f4c2b59e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4c2c14be90 0x7f4c2c132d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59676/0 (socket says 172.21.15.69:59676) 2024-03-25T19:50:25.655 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.655+0000 7f4c2b59e640 1 -- 172.21.15.69:0/1349360508 learned_addr learned my addr 172.21.15.69:0/1349360508 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:25.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.658+0000 7f4c137fe640 10 client.?.objecter ms_handle_connect 0x7f4c2c14be90 2024-03-25T19:50:25.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.658+0000 7f4c137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.669+0000 7f4c137fe640 10 client.?.objecter ms_handle_connect 0x7f4c2c0b4d60 2024-03-25T19:50:25.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.669+0000 7f4c137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.669+0000 7f4c137fe640 10 client.?.objecter ms_handle_connect 0x7f4c2c0b1730 2024-03-25T19:50:25.669 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.669+0000 7f4c137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.675 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.675+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217945420 0 0) 0x7f4c2c13b910 con 0x7f4c2c14be90 2024-03-25T19:50:25.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.675+0000 7f4c137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.675+0000 7f4c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.675+0000 7f4c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.675+0000 7f4c137fe640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:50:25.676 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.675+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c18004eb0 con 0x7f4c2c14be90 2024-03-25T19:50:25.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.681+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137305269 0 0) 0x7f4c2c13a9d0 con 0x7f4c2c0b1730 2024-03-25T19:50:25.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.681+0000 7f4c137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.681+0000 7f4c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.681+0000 7f4c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.681+0000 7f4c137fe640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:50:25.681 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.681+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2c13b910 con 0x7f4c2c0b1730 2024-03-25T19:50:25.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.686+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382920327 0 0) 0x7f4c2c1d7940 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.686+0000 7f4c137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.686+0000 7f4c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.686+0000 7f4c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.686+0000 7f4c137fe640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:50:25.687 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.686+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2c13a9d0 con 0x7f4c2c0b4d60 2024-03-25T19:50:25.693 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.693+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3506616943 0 0) 0x7f4c2c13b910 con 0x7f4c2c0b1730 2024-03-25T19:50:25.693 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.693+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c2c1d7940 con 0x7f4c2c0b1730 2024-03-25T19:50:25.693 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.693+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4c1c001650 con 0x7f4c2c0b1730 2024-03-25T19:50:25.693 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.693+0000 7f4c137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.693 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.693+0000 7f4c137fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:25.693 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.693+0000 7f4c137fe640 10 monclient(hunting): dump: 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:25.694 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.693+0000 7f4c137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1994312043 0 0) 0x7f4c2c1d7940 con 0x7f4c2c0b1730 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 >> v1:172.21.15.169:6789/0 conn(0x7f4c2c0b4d60 legacy=0x7f4c2c1d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 >> v1:172.21.15.69:6789/0 conn(0x7f4c2c14be90 legacy=0x7f4c2c132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 1 monclient: found mon.c 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c13ab60 con 0x7f4c2c0b1730 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 5 monclient: authenticate success, global_id 4640 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 10 monclient: _renew_subs 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:25.698 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c1d8530 con 0x7f4c2c0b1730 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 10 monclient: _renew_subs 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c134d90 con 0x7f4c2c0b1730 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.697+0000 7f4c3109d640 1 librados: init done 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.698+0000 7f4c3109d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.698+0000 7f4c3109d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.698+0000 7f4c3109d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:25.699 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:25.698+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c0b2270 con 0x7f4c2c0b1730 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.005+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c1c0010a0 con 0x7f4c2c0b1730 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.005+0000 7f4c137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.005+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f4c1c004d90 con 0x7f4c2c0b1730 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.005+0000 7f4c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.005+0000 7f4c137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.005+0000 7f4c137fe640 10 monclient: dump: 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:26.006 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:26.007 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:26.007 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:26.007 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:26.007 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:26.007 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:26.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.005+0000 7f4c2959a640 4 set_mon_vals no callback set 2024-03-25T19:50:26.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.006+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f4c1c01ad60 con 0x7f4c2c0b1730 2024-03-25T19:50:26.007 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.006+0000 7f4c2a59c640 10 monclient: get_auth_request con 0x7f4c1806de20 auth_method 0 2024-03-25T19:50:26.008 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.007+0000 7f4c137fe640 10 client.4640.objecter ms_handle_connect 0x7f4c1806de20 2024-03-25T19:50:26.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.012+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f4c1c054820 con 0x7f4c2c0b1730 2024-03-25T19:50:26.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.012+0000 7f4c137fe640 10 client.4640.objecter ms_dispatch 0x7f4c2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:26.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.012+0000 7f4c137fe640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:26.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.012+0000 7f4c137fe640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:26.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.012+0000 7f4c137fe640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-25T19:50:26.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.012+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c1c050330 con 0x7f4c2c0b1730 2024-03-25T19:50:26.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.012+0000 7f4c137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:26.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.012+0000 7f4c137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:26.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.262+0000 7f4c3109d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.263+0000 7f4c3109d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.263+0000 7f4c3109d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:26.263 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.263+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c2c0627b0 con 0x7f4c2c0b1730 2024-03-25T19:50:26.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.301+0000 7f4c137fe640 1 -- 172.21.15.69:0/1349360508 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7f4c1c052260 con 0x7f4c2c0b1730 2024-03-25T19:50:26.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.301+0000 7f4c137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.301+0000 7f4c137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:26.301 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:50:26.301 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 10 librados: watch_flush enter 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 10 librados: watch_flush exit 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 10 client.4640.objecter successfully canceled tick 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 >> v1:172.21.15.169:6800/3072548259 conn(0x7f4c1806de20 legacy=0x7f4c180702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 10 monclient: shutdown 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 >> v1:172.21.15.69:6790/0 conn(0x7f4c2c0b1730 legacy=0x7f4c2c13a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 shutdown_connections 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 >> 172.21.15.69:0/1349360508 conn(0x7f4c2c0617a0 msgr2=0x7f4c2c13bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 shutdown_connections 2024-03-25T19:50:26.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 1 -- 172.21.15.69:0/1349360508 wait complete. 2024-03-25T19:50:26.305 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:26.303+0000 7f4c3109d640 1 librados: shutdown 2024-03-25T19:50:26.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:33.122 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.239+0000 7fce9038c640 10 monclient: get_monmap_and_config 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.239+0000 7fce9038c640 10 monclient: build_initial_monmap 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.239+0000 7fce9038c640 10 monclient: monmap: 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:50:33.240298+0000 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:50:33.240298+0000 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:33.239 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:33.240 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:33.240 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:50:33.240 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:50:33.240 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:33.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.239+0000 7fce9038c640 1 Processor -- start 2024-03-25T19:50:33.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 1 -- start start 2024-03-25T19:50:33.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 10 monclient: init 2024-03-25T19:50:33.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:33.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:33.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 10 monclient(hunting): picked mon.noname-b con 0x7fce880b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:33.241 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 10 monclient(hunting): picked mon.noname-c con 0x7fce880b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:33.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 10 monclient(hunting): picked mon.noname-a con 0x7fce8814be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:33.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce880af940 con 0x7fce8814be90 2024-03-25T19:50:33.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce880b0a70 con 0x7fce880b1730 2024-03-25T19:50:33.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce880b0d50 con 0x7fce880b4d60 2024-03-25T19:50:33.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:33.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.240+0000 7fce9038c640 10 monclient(hunting): authenticate will time out at 5445.509596s 2024-03-25T19:50:33.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.249+0000 7fce8e128640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7fce880b4d60 0x7fce8813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:45970/0 (socket says 172.21.15.69:45970) 2024-03-25T19:50:33.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.249+0000 7fce8e128640 1 -- 172.21.15.69:0/1188831781 learned_addr learned my addr 172.21.15.69:0/1188831781 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:33.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.257+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967616018 0 0) 0x7fce880b0d50 con 0x7fce880b4d60 2024-03-25T19:50:33.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.257+0000 7fce8d126640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.257+0000 7fce8d126640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.257 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.257+0000 7fce8d126640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.258 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.257+0000 7fce8d126640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:50:33.258 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.257+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce74003520 con 0x7fce880b4d60 2024-03-25T19:50:33.265 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.265+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403972776 0 0) 0x7fce880b0a70 con 0x7fce880b1730 2024-03-25T19:50:33.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.266+0000 7fce8d126640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.266+0000 7fce8d126640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.266+0000 7fce8d126640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.266+0000 7fce8d126640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:50:33.266 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.266+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce880b0d50 con 0x7fce880b1730 2024-03-25T19:50:33.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.272+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194242897 0 0) 0x7fce880af940 con 0x7fce8814be90 2024-03-25T19:50:33.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.272+0000 7fce8d126640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.272+0000 7fce8d126640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.272+0000 7fce8d126640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.272+0000 7fce8d126640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:50:33.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.273+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce880b0a70 con 0x7fce8814be90 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604500636 0 0) 0x7fce880b0d50 con 0x7fce880b1730 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 >> v1:172.21.15.69:6790/0 conn(0x7fce880b4d60 legacy=0x7fce8813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 >> v1:172.21.15.69:6789/0 conn(0x7fce8814be90 legacy=0x7fce8814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 10 monclient: _finish_hunting 0 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 1 monclient: found mon.noname-c 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880b1030 con 0x7fce880b1730 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 10 monclient: _finish_auth 0 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 10 monclient: _check_auth_tickets 2024-03-25T19:50:33.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce78001670 con 0x7fce880b1730 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce8d126640 10 monclient: dump: 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:33.282 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:33.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce9038c640 5 monclient: authenticate success, global_id 4642 2024-03-25T19:50:33.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.280+0000 7fce9038c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.296+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce780017d0 con 0x7fce880b1730 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.296+0000 7fce8d126640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.296+0000 7fce8d126640 1 -- 172.21.15.69:0/1188831781 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce78001be0 con 0x7fce880b1730 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.296+0000 7fce8d126640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.296+0000 7fce8d126640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.296+0000 7fce8d126640 10 monclient: dump: 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.297 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:33.298 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:33.298 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:33.298 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:33.298 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:33.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.296+0000 7fce9038c640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:33.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.296+0000 7fce9038c640 4 set_mon_vals no callback set 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.297+0000 7fce9038c640 10 monclient: shutdown 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.297+0000 7fce9038c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.297+0000 7fce9038c640 1 -- 172.21.15.69:0/1188831781 >> v1:172.21.15.169:6789/0 conn(0x7fce880b1730 legacy=0x7fce880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.297+0000 7fce9038c640 1 -- 172.21.15.69:0/1188831781 shutdown_connections 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.297+0000 7fce9038c640 1 -- 172.21.15.69:0/1188831781 >> 172.21.15.69:0/1188831781 conn(0x7fce880617a0 msgr2=0x7fce88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.297+0000 7fce9038c640 1 -- 172.21.15.69:0/1188831781 shutdown_connections 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.297+0000 7fce9038c640 1 -- 172.21.15.69:0/1188831781 wait complete. 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 10 monclient: build_initial_monmap 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 10 monclient: monmap: 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:50:33.300 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 1 librados: starting msgr at 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 1 librados: starting objecter 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 1 Processor -- start 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 1 -- start start 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 1 librados: setting wanted keys 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 1 librados: calling monclient init 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.298+0000 7fce9038c640 10 monclient: init 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 10 monclient(hunting): picked mon.noname-a con 0x7fce880b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 10 monclient(hunting): picked mon.noname-c con 0x7fce880b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:33.301 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 10 monclient(hunting): picked mon.noname-b con 0x7fce8814be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:33.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce881d7850 con 0x7fce880b1730 2024-03-25T19:50:33.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce881382f0 con 0x7fce8814be90 2024-03-25T19:50:33.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce881d83f0 con 0x7fce880b4d60 2024-03-25T19:50:33.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:33.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.299+0000 7fce9038c640 10 monclient(hunting): authenticate will time out at 5445.568146s 2024-03-25T19:50:33.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.311+0000 7fce8e128640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fce880b1730 0x7fce881d7ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:38528/0 (socket says 172.21.15.69:38528) 2024-03-25T19:50:33.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.311+0000 7fce8e128640 1 -- 172.21.15.69:0/3498171259 learned_addr learned my addr 172.21.15.69:0/3498171259 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:33.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.313+0000 7fce7effd640 10 client.?.objecter ms_handle_connect 0x7fce880b1730 2024-03-25T19:50:33.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.313+0000 7fce7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.313+0000 7fce7effd640 10 client.?.objecter ms_handle_connect 0x7fce880b4d60 2024-03-25T19:50:33.313 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.313+0000 7fce7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.316+0000 7fce7effd640 10 client.?.objecter ms_handle_connect 0x7fce8814be90 2024-03-25T19:50:33.316 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.316+0000 7fce7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.324+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663040206 0 0) 0x7fce881382f0 con 0x7fce8814be90 2024-03-25T19:50:33.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.324+0000 7fce7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.324+0000 7fce7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.324+0000 7fce7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.324+0000 7fce7effd640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:50:33.324 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.324+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6c004eb0 con 0x7fce8814be90 2024-03-25T19:50:33.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.328+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905736373 0 0) 0x7fce881d83f0 con 0x7fce880b4d60 2024-03-25T19:50:33.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.328+0000 7fce7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.328+0000 7fce7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.328+0000 7fce7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.328+0000 7fce7effd640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:50:33.328 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.328+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce881382f0 con 0x7fce880b4d60 2024-03-25T19:50:33.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.333+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 170929022 0 0) 0x7fce6c004eb0 con 0x7fce8814be90 2024-03-25T19:50:33.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.333+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce881d83f0 con 0x7fce8814be90 2024-03-25T19:50:33.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.333+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce80001bd0 con 0x7fce8814be90 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.333+0000 7fce7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.333+0000 7fce7effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.333+0000 7fce7effd640 10 monclient(hunting): dump: 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:33.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.333+0000 7fce7effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.338+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3545023728 0 0) 0x7fce881382f0 con 0x7fce880b4d60 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.338+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce6c004eb0 con 0x7fce880b4d60 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.338+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce78001ae0 con 0x7fce880b4d60 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.338+0000 7fce7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.338+0000 7fce7effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.338+0000 7fce7effd640 10 monclient(hunting): dump: 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:33.338 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:33.339 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:33.339 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:33.339 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:33.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.341+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450794866 0 0) 0x7fce881d7850 con 0x7fce880b1730 2024-03-25T19:50:33.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.341+0000 7fce7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.341+0000 7fce7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.341+0000 7fce7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.341+0000 7fce7effd640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:50:33.342 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.341+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce881382f0 con 0x7fce880b1730 2024-03-25T19:50:33.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1352531345 0 0) 0x7fce6c004eb0 con 0x7fce880b4d60 2024-03-25T19:50:33.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 >> v1:172.21.15.169:6789/0 conn(0x7fce8814be90 legacy=0x7fce8812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 >> v1:172.21.15.69:6789/0 conn(0x7fce880b1730 legacy=0x7fce881d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:33.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 1 monclient: found mon.c 2024-03-25T19:50:33.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce88135c90 con 0x7fce880b4d60 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 10 monclient: _finish_auth 0 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce9038c640 5 monclient: authenticate success, global_id 4646 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce9038c640 10 monclient: _renew_subs 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce9038c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.353+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce88136e80 con 0x7fce880b4d60 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.354+0000 7fce9038c640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.354+0000 7fce9038c640 10 monclient: _renew_subs 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.354+0000 7fce9038c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.354+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce88143ae0 con 0x7fce880b4d60 2024-03-25T19:50:33.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.354+0000 7fce9038c640 1 librados: init done 2024-03-25T19:50:33.356 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.355+0000 7fce9038c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.356 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.355+0000 7fce9038c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.356 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.355+0000 7fce9038c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:33.356 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.355+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce880627b0 con 0x7fce880b4d60 2024-03-25T19:50:33.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.375+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce78001210 con 0x7fce880b4d60 2024-03-25T19:50:33.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.375+0000 7fce7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.375+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce78003a40 con 0x7fce880b4d60 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.375+0000 7fce7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.375+0000 7fce7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.375+0000 7fce7effd640 10 monclient: dump: 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:33.376 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:33.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.375+0000 7fce8c925640 4 set_mon_vals no callback set 2024-03-25T19:50:33.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.376+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fce78019a10 con 0x7fce880b4d60 2024-03-25T19:50:33.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.377+0000 7fce8e128640 10 monclient: get_auth_request con 0x7fce6c06dd70 auth_method 0 2024-03-25T19:50:33.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.377+0000 7fce7effd640 10 client.4646.objecter ms_handle_connect 0x7fce6c06dd70 2024-03-25T19:50:33.379 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.379+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fce78053660 con 0x7fce880b4d60 2024-03-25T19:50:33.379 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.379+0000 7fce7effd640 10 client.4646.objecter ms_dispatch 0x7fce88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:33.379 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.379+0000 7fce7effd640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:33.379 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.379+0000 7fce7effd640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:33.380 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.379+0000 7fce7effd640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-25T19:50:33.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.665+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce7804efa0 con 0x7fce880b4d60 2024-03-25T19:50:33.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.666+0000 7fce7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.666 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.666+0000 7fce7effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:33.857 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:11.821 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.880 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.880+0000 7fce9038c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.880+0000 7fce9038c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.880+0000 7fce9038c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:33.881 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.880+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce88069f40 con 0x7fce880b4d60 2024-03-25T19:50:33.911 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:11.875 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.928 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:11.891 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.964 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:11.928 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.973 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:11.927 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.987 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:11.951 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.990 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:11.945 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.992+0000 7fce7effd640 1 -- 172.21.15.69:0/3498171259 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fce780510a0 con 0x7fce880b4d60 2024-03-25T19:50:33.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.992+0000 7fce7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.992+0000 7fce7effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:33.992 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:50:33.992 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:33.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 10 librados: watch_flush enter 2024-03-25T19:50:33.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 10 librados: watch_flush exit 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 10 client.4646.objecter successfully canceled tick 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 >> v1:172.21.15.169:6800/3072548259 conn(0x7fce6c06dd70 legacy=0x7fce6c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 10 monclient: shutdown 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 >> v1:172.21.15.69:6790/0 conn(0x7fce880b4d60 legacy=0x7fce88136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 shutdown_connections 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 >> 172.21.15.69:0/3498171259 conn(0x7fce880617a0 msgr2=0x7fce8813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 shutdown_connections 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 1 -- 172.21.15.69:0/3498171259 wait complete. 2024-03-25T19:50:33.996 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:33.995+0000 7fce9038c640 1 librados: shutdown 2024-03-25T19:50:34.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:34.015 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:11.969 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:34.022 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:11.986 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:40.813 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.913+0000 7fc2f68f6640 10 monclient: get_monmap_and_config 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.913+0000 7fc2f68f6640 10 monclient: build_initial_monmap 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.913+0000 7fc2f68f6640 10 monclient: monmap: 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:50:40.914269+0000 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:50:40.914269+0000 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:40.913 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:40.914 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:50:40.914 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:50:40.914 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:40.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 1 Processor -- start 2024-03-25T19:50:40.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 1 -- start start 2024-03-25T19:50:40.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 10 monclient: init 2024-03-25T19:50:40.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:40.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:40.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f00705b0 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:40.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f01402d0 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:40.915 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f0149ae0 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:40.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f00718c0 con 0x7fc2f01402d0 2024-03-25T19:50:40.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f00714c0 con 0x7fc2f0149ae0 2024-03-25T19:50:40.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f0072b50 con 0x7fc2f00705b0 2024-03-25T19:50:40.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:40.916 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.914+0000 7fc2f68f6640 10 monclient(hunting): authenticate will time out at 5453.183313s 2024-03-25T19:50:40.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.923+0000 7fc2f4e93640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc2f0149ae0 0x7fc2f014bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37228/0 (socket says 172.21.15.69:37228) 2024-03-25T19:50:40.923 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.923+0000 7fc2f4e93640 1 -- 172.21.15.69:0/1279985199 learned_addr learned my addr 172.21.15.69:0/1279985199 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:40.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.930+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124586421 0 0) 0x7fc2f00714c0 con 0x7fc2f0149ae0 2024-03-25T19:50:40.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.930+0000 7fc2eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.930+0000 7fc2eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.930+0000 7fc2eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.930+0000 7fc2eeffd640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:50:40.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.930+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d8003520 con 0x7fc2f0149ae0 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220444839 0 0) 0x7fc2d8003520 con 0x7fc2f0149ae0 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 >> v1:172.21.15.69:6790/0 conn(0x7fc2f00705b0 legacy=0x7fc2f013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 >> v1:172.21.15.69:6789/0 conn(0x7fc2f01402d0 legacy=0x7fc2f0140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 1 monclient: found mon.noname-c 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f013dd10 con 0x7fc2f0149ae0 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2f68f6640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:50:40.937 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc2e8001a10 con 0x7fc2f0149ae0 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2f68f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2eeffd640 10 monclient: dump: 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:40.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.936+0000 7fc2f68f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:40.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.953+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2e8001be0 con 0x7fc2f0149ae0 2024-03-25T19:50:40.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.953+0000 7fc2eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:40.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.953+0000 7fc2eeffd640 1 -- 172.21.15.69:0/1279985199 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc2e8002030 con 0x7fc2f0149ae0 2024-03-25T19:50:40.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.953+0000 7fc2f68f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:40.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.953+0000 7fc2f68f6640 4 set_mon_vals no callback set 2024-03-25T19:50:40.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.953+0000 7fc2eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.953+0000 7fc2eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.953+0000 7fc2eeffd640 10 monclient: dump: 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:40.954 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:40.955 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.954+0000 7fc2f68f6640 10 monclient: shutdown 2024-03-25T19:50:40.955 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.954+0000 7fc2f68f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:40.955 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.954+0000 7fc2f68f6640 1 -- 172.21.15.69:0/1279985199 >> v1:172.21.15.169:6789/0 conn(0x7fc2f0149ae0 legacy=0x7fc2f014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.954+0000 7fc2f68f6640 1 -- 172.21.15.69:0/1279985199 shutdown_connections 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.954+0000 7fc2f68f6640 1 -- 172.21.15.69:0/1279985199 >> 172.21.15.69:0/1279985199 conn(0x7fc2f00b5d10 msgr2=0x7fc2f00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.954+0000 7fc2f68f6640 1 -- 172.21.15.69:0/1279985199 shutdown_connections 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.954+0000 7fc2f68f6640 1 -- 172.21.15.69:0/1279985199 wait complete. 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.955+0000 7fc2f68f6640 10 monclient: build_initial_monmap 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.955+0000 7fc2f68f6640 10 monclient: monmap: 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:40.956 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:50:40.957 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:50:40.957 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:40.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.955+0000 7fc2f68f6640 1 librados: starting msgr at 2024-03-25T19:50:40.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.955+0000 7fc2f68f6640 1 librados: starting objecter 2024-03-25T19:50:40.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.955+0000 7fc2f68f6640 1 Processor -- start 2024-03-25T19:50:40.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 1 -- start start 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 1 librados: setting wanted keys 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 1 librados: calling monclient init 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 10 monclient: init 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f00705b0 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f01402d0 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f0149ae0 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f01443e0 con 0x7fc2f01402d0 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f0143b40 con 0x7fc2f0149ae0 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f0141750 con 0x7fc2f00705b0 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:40.958 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.956+0000 7fc2f68f6640 10 monclient(hunting): authenticate will time out at 5453.225452s 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.967+0000 7fc2ef7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc2f01402d0 0x7fc2f01d5a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42462/0 (socket says 172.21.15.69:42462) 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.967+0000 7fc2ef7fe640 1 -- 172.21.15.69:0/717605698 learned_addr learned my addr 172.21.15.69:0/717605698 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:40.969 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.969+0000 7fc2ecff9640 10 client.?.objecter ms_handle_connect 0x7fc2f01402d0 2024-03-25T19:50:40.969 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.969+0000 7fc2ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.971+0000 7fc2ecff9640 10 client.?.objecter ms_handle_connect 0x7fc2f0149ae0 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.971+0000 7fc2ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.975+0000 7fc2ecff9640 10 client.?.objecter ms_handle_connect 0x7fc2f00705b0 2024-03-25T19:50:40.975 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.975+0000 7fc2ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.981+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169121951 0 0) 0x7fc2f01443e0 con 0x7fc2f01402d0 2024-03-25T19:50:40.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.981+0000 7fc2ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.981+0000 7fc2ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.981+0000 7fc2ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.981+0000 7fc2ecff9640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:50:40.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.981+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc004eb0 con 0x7fc2f01402d0 2024-03-25T19:50:40.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.991+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986117632 0 0) 0x7fc2f0141750 con 0x7fc2f00705b0 2024-03-25T19:50:40.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.991+0000 7fc2ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.991+0000 7fc2ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.991+0000 7fc2ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.991+0000 7fc2ecff9640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:50:40.991 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.991+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f01443e0 con 0x7fc2f00705b0 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.992+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2301483841 0 0) 0x7fc2dc004eb0 con 0x7fc2f01402d0 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.992+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2f0141750 con 0x7fc2f01402d0 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.992+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc2e4001a30 con 0x7fc2f01402d0 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.992+0000 7fc2ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.993+0000 7fc2ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.993+0000 7fc2ecff9640 10 monclient(hunting): dump: 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.993 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:40.994 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:40.994 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:40.994 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:40.994 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:40.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.993+0000 7fc2ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:40.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.995+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152728349 0 0) 0x7fc2f0143b40 con 0x7fc2f0149ae0 2024-03-25T19:50:40.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.995+0000 7fc2ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.995+0000 7fc2ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.995+0000 7fc2ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.995+0000 7fc2ecff9640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:50:40.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:40.995+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc004eb0 con 0x7fc2f0149ae0 2024-03-25T19:50:41.012 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.012+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448406580 0 0) 0x7fc2f01443e0 con 0x7fc2f00705b0 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.012+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2f0143b40 con 0x7fc2f00705b0 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.012+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc2e0001810 con 0x7fc2f00705b0 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.012+0000 7fc2ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.012+0000 7fc2ecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.012+0000 7fc2ecff9640 10 monclient(hunting): dump: 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:41.014 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:41.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 42992819 0 0) 0x7fc2f0141750 con 0x7fc2f01402d0 2024-03-25T19:50:41.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 >> v1:172.21.15.69:6790/0 conn(0x7fc2f00705b0 legacy=0x7fc2f01d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.021 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 >> v1:172.21.15.169:6789/0 conn(0x7fc2f0149ae0 legacy=0x7fc2f01dfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 1 monclient: found mon.a 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f01418e0 con 0x7fc2f01402d0 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 5 monclient: authenticate success, global_id 4680 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 10 monclient: _renew_subs 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f0141ba0 con 0x7fc2f01402d0 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 10 client.4680.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:41.022 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 10 monclient: _renew_subs 2024-03-25T19:50:41.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:50:41.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f01424d0 con 0x7fc2f01402d0 2024-03-25T19:50:41.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.021+0000 7fc2f68f6640 1 librados: init done 2024-03-25T19:50:41.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.022+0000 7fc2f68f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:41.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.022+0000 7fc2f68f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:41.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.022+0000 7fc2f68f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:50:41.023 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.022+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0144240 con 0x7fc2f01402d0 2024-03-25T19:50:41.332 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.332+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2e4001c70 con 0x7fc2f01402d0 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.332+0000 7fc2ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.332+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fc2e40038c0 con 0x7fc2f01402d0 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.332+0000 7fc2ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.332+0000 7fc2ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.332+0000 7fc2ecff9640 10 monclient: dump: 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:41.333 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:41.334 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:41.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.332+0000 7fc2ee7fc640 4 set_mon_vals no callback set 2024-03-25T19:50:41.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.333+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fc2e4003aa0 con 0x7fc2f01402d0 2024-03-25T19:50:41.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.334+0000 7fc2effff640 10 monclient: get_auth_request con 0x7fc2dc06dfb0 auth_method 0 2024-03-25T19:50:41.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.334+0000 7fc2ecff9640 10 client.4680.objecter ms_handle_connect 0x7fc2dc06dfb0 2024-03-25T19:50:41.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.338+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7fc2e40546f0 con 0x7fc2f01402d0 2024-03-25T19:50:41.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.338+0000 7fc2ecff9640 10 client.4680.objecter ms_dispatch 0x7fc2f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:41.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.338+0000 7fc2ecff9640 3 client.4680.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:41.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.338+0000 7fc2ecff9640 3 client.4680.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:41.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.338+0000 7fc2ecff9640 20 client.4680.objecter dump_active .. 0 homeless 2024-03-25T19:50:41.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.338+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2e4050200 con 0x7fc2f01402d0 2024-03-25T19:50:41.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.339+0000 7fc2ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:41.339 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.339+0000 7fc2ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:41.593 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.593+0000 7fc2f68f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.593 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.593+0000 7fc2f68f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.593 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.593+0000 7fc2f68f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:50:41.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.593+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2f00627b0 con 0x7fc2f01402d0 2024-03-25T19:50:41.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.630+0000 7fc2ecff9640 1 -- 172.21.15.69:0/717605698 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2560494771) 0x7fc2e4052130 con 0x7fc2f01402d0 2024-03-25T19:50:41.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.630+0000 7fc2ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.630+0000 7fc2ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:41.631 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:50:41.631 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:40:50.020605+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:50.020605+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:41.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.632+0000 7fc2f68f6640 10 librados: watch_flush enter 2024-03-25T19:50:41.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 10 librados: watch_flush exit 2024-03-25T19:50:41.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 20 client.4680.objecter shutdown clearing up homeless session... 2024-03-25T19:50:41.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 10 client.4680.objecter successfully canceled tick 2024-03-25T19:50:41.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 >> v1:172.21.15.169:6800/3072548259 conn(0x7fc2dc06dfb0 legacy=0x7fc2dc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.633 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 10 monclient: shutdown 2024-03-25T19:50:41.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:41.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 >> v1:172.21.15.69:6789/0 conn(0x7fc2f01402d0 legacy=0x7fc2f01d5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 shutdown_connections 2024-03-25T19:50:41.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 >> 172.21.15.69:0/717605698 conn(0x7fc2f00b5d10 msgr2=0x7fc2f0140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:41.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 shutdown_connections 2024-03-25T19:50:41.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 1 -- 172.21.15.69:0/717605698 wait complete. 2024-03-25T19:50:41.634 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:41.633+0000 7fc2f68f6640 1 librados: shutdown 2024-03-25T19:50:41.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:48.452 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:48.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.571+0000 7f96f7d65640 10 monclient: get_monmap_and_config 2024-03-25T19:50:48.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.571+0000 7f96f7d65640 10 monclient: build_initial_monmap 2024-03-25T19:50:48.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.571+0000 7f96f7d65640 10 monclient: monmap: 2024-03-25T19:50:48.570 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:48.570 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:48.571 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:50:48.571505+0000 2024-03-25T19:50:48.571 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:50:48.571505+0000 2024-03-25T19:50:48.571 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:48.571 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:48.571 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:48.571 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:50:48.571 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:50:48.571 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:48.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.571+0000 7f96f7d65640 1 Processor -- start 2024-03-25T19:50:48.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.572+0000 7f96f7d65640 1 -- start start 2024-03-25T19:50:48.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.572+0000 7f96f7d65640 10 monclient: init 2024-03-25T19:50:48.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:48.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:48.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 10 monclient(hunting): picked mon.noname-c con 0x7f96f00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:48.572 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 10 monclient(hunting): picked mon.noname-a con 0x7f96f00b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:48.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 10 monclient(hunting): picked mon.noname-b con 0x7f96f014be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:48.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f00af940 con 0x7f96f00b1730 2024-03-25T19:50:48.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f00b0a70 con 0x7f96f00b4d60 2024-03-25T19:50:48.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f00b0d50 con 0x7f96f014be90 2024-03-25T19:50:48.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:48.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.573+0000 7f96f7d65640 10 monclient(hunting): authenticate will time out at 5460.840962s 2024-03-25T19:50:48.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.590+0000 7f96f5300640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f96f00b1730 0x7f96f00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47096/0 (socket says 172.21.15.69:47096) 2024-03-25T19:50:48.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.590+0000 7f96f5300640 1 -- 172.21.15.69:0/642598667 learned_addr learned my addr 172.21.15.69:0/642598667 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:48.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.611+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754436211 0 0) 0x7f96f00b0a70 con 0x7f96f00b4d60 2024-03-25T19:50:48.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.611+0000 7f96f4aff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.611+0000 7f96f4aff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.611+0000 7f96f4aff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.610 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.611+0000 7f96f4aff640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:50:48.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.611+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96dc003520 con 0x7f96f00b4d60 2024-03-25T19:50:48.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278316877 0 0) 0x7f96f00b0d50 con 0x7f96f014be90 2024-03-25T19:50:48.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.611 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:50:48.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f00b0a70 con 0x7f96f014be90 2024-03-25T19:50:48.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947589048 0 0) 0x7f96f00af940 con 0x7f96f00b1730 2024-03-25T19:50:48.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:50:48.612 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.612+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f00b0d50 con 0x7f96f00b1730 2024-03-25T19:50:48.617 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 193446960 0 0) 0x7f96dc003520 con 0x7f96f00b4d60 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 >> v1:172.21.15.69:6790/0 conn(0x7f96f014be90 legacy=0x7f96f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 >> v1:172.21.15.69:6789/0 conn(0x7f96f00b1730 legacy=0x7f96f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 1 monclient: found mon.noname-c 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00b1030 con 0x7f96f00b4d60 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 10 monclient: _finish_auth 0 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f96ec001690 con 0x7f96f00b4d60 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f4aff640 10 monclient: dump: 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f7d65640 5 monclient: authenticate success, global_id 4654 2024-03-25T19:50:48.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.618+0000 7f96f7d65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:48.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.641+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96ec0017f0 con 0x7f96f00b4d60 2024-03-25T19:50:48.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.641+0000 7f96f4aff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:48.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.641+0000 7f96f4aff640 1 -- 172.21.15.69:0/642598667 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f96ec001c00 con 0x7f96f00b4d60 2024-03-25T19:50:48.640 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.641+0000 7f96f7d65640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.641+0000 7f96f7d65640 4 set_mon_vals no callback set 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.641+0000 7f96f4aff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.641+0000 7f96f4aff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.641+0000 7f96f4aff640 10 monclient: dump: 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:48.641 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:48.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.642+0000 7f96f7d65640 10 monclient: shutdown 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.642+0000 7f96f7d65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.642+0000 7f96f7d65640 1 -- 172.21.15.69:0/642598667 >> v1:172.21.15.169:6789/0 conn(0x7f96f00b4d60 legacy=0x7f96f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.642+0000 7f96f7d65640 1 -- 172.21.15.69:0/642598667 shutdown_connections 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.642+0000 7f96f7d65640 1 -- 172.21.15.69:0/642598667 >> 172.21.15.69:0/642598667 conn(0x7f96f00617a0 msgr2=0x7f96f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.642+0000 7f96f7d65640 1 -- 172.21.15.69:0/642598667 shutdown_connections 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.642+0000 7f96f7d65640 1 -- 172.21.15.69:0/642598667 wait complete. 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 10 monclient: build_initial_monmap 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 10 monclient: monmap: 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:50:48.644 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 1 librados: starting msgr at 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 1 librados: starting objecter 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 1 Processor -- start 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 1 -- start start 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 1 librados: setting wanted keys 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 1 librados: calling monclient init 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.643+0000 7f96f7d65640 10 monclient: init 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 10 monclient(hunting): picked mon.noname-c con 0x7f96f00b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 10 monclient(hunting): picked mon.noname-b con 0x7f96f00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 10 monclient(hunting): picked mon.noname-a con 0x7f96f014be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:48.645 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f01d84a0 con 0x7f96f014be90 2024-03-25T19:50:48.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f013ac40 con 0x7f96f00b4d60 2024-03-25T19:50:48.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f01d7590 con 0x7f96f00b1730 2024-03-25T19:50:48.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:48.646 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.644+0000 7f96f7d65640 10 monclient(hunting): authenticate will time out at 5460.911847s 2024-03-25T19:50:48.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.655+0000 7f96f5b01640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f96f00b1730 0x7f96f013bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:42860/0 (socket says 172.21.15.69:42860) 2024-03-25T19:50:48.654 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.655+0000 7f96f5b01640 1 -- 172.21.15.69:0/828346834 learned_addr learned my addr 172.21.15.69:0/828346834 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:48.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.658+0000 7f96e67fc640 10 client.?.objecter ms_handle_connect 0x7f96f00b1730 2024-03-25T19:50:48.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.658+0000 7f96e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.668+0000 7f96e67fc640 10 client.?.objecter ms_handle_connect 0x7f96f00b4d60 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.668+0000 7f96e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.670 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.671+0000 7f96e67fc640 10 client.?.objecter ms_handle_connect 0x7f96f014be90 2024-03-25T19:50:48.670 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.671+0000 7f96e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.673+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405738933 0 0) 0x7f96f01d7590 con 0x7f96f00b1730 2024-03-25T19:50:48.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.673+0000 7f96e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.673+0000 7f96e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.673+0000 7f96e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.673+0000 7f96e67fc640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:50:48.673 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.673+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d4004eb0 con 0x7f96f00b1730 2024-03-25T19:50:48.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.679+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122106488 0 0) 0x7f96f013ac40 con 0x7f96f00b4d60 2024-03-25T19:50:48.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.679+0000 7f96e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.679+0000 7f96e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.679+0000 7f96e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.679+0000 7f96e67fc640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:50:48.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.679+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f01d7590 con 0x7f96f00b4d60 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.703+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1991549729 0 0) 0x7f96d4004eb0 con 0x7f96f00b1730 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.703+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96f013ac40 con 0x7f96f00b1730 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.703+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f96ec001bc0 con 0x7f96f00b1730 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.703+0000 7f96e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.703+0000 7f96e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.703+0000 7f96e67fc640 10 monclient(hunting): dump: 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:48.702 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.703+0000 7f96e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.704+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070349592 0 0) 0x7f96f01d7590 con 0x7f96f00b4d60 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.704+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96d4004eb0 con 0x7f96f00b4d60 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.704+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f96e0001870 con 0x7f96f00b4d60 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.704+0000 7f96e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.704+0000 7f96e67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.704+0000 7f96e67fc640 10 monclient(hunting): dump: 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:48.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3575905877 0 0) 0x7f96d4004eb0 con 0x7f96f00b4d60 2024-03-25T19:50:48.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 >> v1:172.21.15.69:6790/0 conn(0x7f96f00b1730 legacy=0x7f96f013bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.719 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 >> v1:172.21.15.69:6789/0 conn(0x7f96f014be90 legacy=0x7f96f0139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 1 monclient: found mon.b 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f0142ae0 con 0x7f96f00b4d60 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 10 monclient: _renew_subs 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f0139a10 con 0x7f96f00b4d60 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 10 monclient: _renew_subs 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f013b240 con 0x7f96f00b4d60 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.720+0000 7f96f7d65640 1 librados: init done 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.721+0000 7f96f7d65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.721+0000 7f96f7d65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.721+0000 7f96f7d65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:48.721+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f013b4f0 con 0x7f96f00b4d60 2024-03-25T19:50:49.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.076+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96e0001cc0 con 0x7f96f00b4d60 2024-03-25T19:50:49.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.076+0000 7f96e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:49.075 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.076+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f96e0004a40 con 0x7f96f00b4d60 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.076+0000 7f96e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.076+0000 7f96e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.076+0000 7f96e67fc640 10 monclient: dump: 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:49.076 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.076+0000 7f96e7fff640 4 set_mon_vals no callback set 2024-03-25T19:50:49.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.077+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f96e0004c60 con 0x7f96f00b4d60 2024-03-25T19:50:49.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.078+0000 7f96f5b01640 10 monclient: get_auth_request con 0x7f96d406de80 auth_method 0 2024-03-25T19:50:49.077 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.078+0000 7f96e67fc640 10 client.4657.objecter ms_handle_connect 0x7f96d406de80 2024-03-25T19:50:49.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.082+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2070511898 0 0) 0x7f96e0054700 con 0x7f96f00b4d60 2024-03-25T19:50:49.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.082+0000 7f96e67fc640 10 client.4657.objecter ms_dispatch 0x7f96f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:49.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.082+0000 7f96e67fc640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:49.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.082+0000 7f96e67fc640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:49.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.082+0000 7f96e67fc640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T19:50:49.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.083+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96e0050210 con 0x7f96f00b4d60 2024-03-25T19:50:49.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.083+0000 7f96e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:49.082 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.083+0000 7f96e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:49.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.310+0000 7f96f7d65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:49.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.310+0000 7f96f7d65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:49.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.310+0000 7f96f7d65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:50:49.310 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.310+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96f00627b0 con 0x7f96f00b4d60 2024-03-25T19:50:49.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.346+0000 7f96e67fc640 1 -- 172.21.15.69:0/828346834 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7f96e0052140 con 0x7f96f00b4d60 2024-03-25T19:50:49.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.346+0000 7f96e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:49.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.346+0000 7f96e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:49.346 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:50:49.346 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:49.348 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 10 librados: watch_flush enter 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 10 librados: watch_flush exit 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 10 client.4657.objecter successfully canceled tick 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 >> v1:172.21.15.169:6800/3072548259 conn(0x7f96d406de80 legacy=0x7f96d4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 10 monclient: shutdown 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 >> v1:172.21.15.169:6789/0 conn(0x7f96f00b4d60 legacy=0x7f96f01d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 shutdown_connections 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 >> 172.21.15.69:0/828346834 conn(0x7f96f00617a0 msgr2=0x7f96f00b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 shutdown_connections 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 1 -- 172.21.15.69:0/828346834 wait complete. 2024-03-25T19:50:49.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:49.349+0000 7f96f7d65640 1 librados: shutdown 2024-03-25T19:50:49.359 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:53.837 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:31.801 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:53.843 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:31.807 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:53.870 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:31.834 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:53.902 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:31.856 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:53.915 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:31.869 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:53.919 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:31.883 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:53.929 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:31.893 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:53.934 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:31.889 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:53.962 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:31.926 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:56.165 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:56.280 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.280+0000 7ff36920f640 10 monclient: get_monmap_and_config 2024-03-25T19:50:56.280 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.280+0000 7ff36920f640 10 monclient: build_initial_monmap 2024-03-25T19:50:56.280 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.280+0000 7ff36920f640 10 monclient: monmap: 2024-03-25T19:50:56.280 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:56.280 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:56.280 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:50:56.281684+0000 2024-03-25T19:50:56.281 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:50:56.281684+0000 2024-03-25T19:50:56.281 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:56.281 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:56.281 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:56.281 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:50:56.281 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:50:56.281 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:56.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 1 Processor -- start 2024-03-25T19:50:56.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 1 -- start start 2024-03-25T19:50:56.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 10 monclient: init 2024-03-25T19:50:56.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 10 monclient(hunting): picked mon.noname-c con 0x7ff3640b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 10 monclient(hunting): picked mon.noname-b con 0x7ff3640b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 10 monclient(hunting): picked mon.noname-a con 0x7ff36414be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.281+0000 7ff36920f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3640af940 con 0x7ff36414be90 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.282+0000 7ff36920f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3640b0a70 con 0x7ff3640b4d60 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.282+0000 7ff36920f640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3640b0d50 con 0x7ff3640b1730 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.282+0000 7ff36920f640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:56.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.282+0000 7ff36920f640 10 monclient(hunting): authenticate will time out at 5468.550643s 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.293+0000 7ff36359e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff36414be90 0x7ff36414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60958/0 (socket says 172.21.15.69:60958) 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.293+0000 7ff36359e640 1 -- 172.21.15.69:0/1824132202 learned_addr learned my addr 172.21.15.69:0/1824132202 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:56.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.302+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753220239 0 0) 0x7ff3640af940 con 0x7ff36414be90 2024-03-25T19:50:56.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.302+0000 7ff361d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.302 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.302+0000 7ff361d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.302+0000 7ff361d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.302+0000 7ff361d9b640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.302+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff34c003520 con 0x7ff36414be90 2024-03-25T19:50:56.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.321+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308093584 0 0) 0x7ff3640b0a70 con 0x7ff3640b4d60 2024-03-25T19:50:56.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.321+0000 7ff361d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.321+0000 7ff361d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.321 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.321+0000 7ff361d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.321+0000 7ff361d9b640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:50:56.322 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.321+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3640af940 con 0x7ff3640b4d60 2024-03-25T19:50:56.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756691568 0 0) 0x7ff34c003520 con 0x7ff36414be90 2024-03-25T19:50:56.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 >> v1:172.21.15.69:6790/0 conn(0x7ff3640b1730 legacy=0x7ff3640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 >> v1:172.21.15.169:6789/0 conn(0x7ff3640b4d60 legacy=0x7ff36413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:50:56.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 1 monclient: found mon.noname-a 2024-03-25T19:50:56.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:56.325 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3640b1030 con 0x7ff36414be90 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 10 monclient: _finish_auth 0 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff35c001a10 con 0x7ff36414be90 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff361d9b640 10 monclient: dump: 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.326 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:56.327 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:56.327 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:56.327 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:56.327 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:56.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff36920f640 5 monclient: authenticate success, global_id 4689 2024-03-25T19:50:56.327 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.325+0000 7ff36920f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:56.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.340+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff35c001b70 con 0x7ff36414be90 2024-03-25T19:50:56.340 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.340+0000 7ff361d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.340+0000 7ff361d9b640 1 -- 172.21.15.69:0/1824132202 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff35c001f80 con 0x7ff36414be90 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.340+0000 7ff36920f640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.340+0000 7ff36920f640 4 set_mon_vals no callback set 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.340+0000 7ff361d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.340+0000 7ff361d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.340+0000 7ff361d9b640 10 monclient: dump: 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:56.341 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.341+0000 7ff36920f640 10 monclient: shutdown 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.341+0000 7ff36920f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.341+0000 7ff36920f640 1 -- 172.21.15.69:0/1824132202 >> v1:172.21.15.69:6789/0 conn(0x7ff36414be90 legacy=0x7ff36414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.341+0000 7ff36920f640 1 -- 172.21.15.69:0/1824132202 shutdown_connections 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.341+0000 7ff36920f640 1 -- 172.21.15.69:0/1824132202 >> 172.21.15.69:0/1824132202 conn(0x7ff3640617a0 msgr2=0x7ff364061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.341+0000 7ff36920f640 1 -- 172.21.15.69:0/1824132202 shutdown_connections 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.341+0000 7ff36920f640 1 -- 172.21.15.69:0/1824132202 wait complete. 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.342+0000 7ff36920f640 10 monclient: build_initial_monmap 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.342+0000 7ff36920f640 10 monclient: monmap: 2024-03-25T19:50:56.343 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 librados: starting msgr at 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 librados: starting objecter 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 Processor -- start 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 -- start start 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 librados: setting wanted keys 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 librados: calling monclient init 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 10 monclient: init 2024-03-25T19:50:56.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:56.345 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 10 monclient(hunting): picked mon.noname-a con 0x7ff3640b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 10 monclient(hunting): picked mon.noname-b con 0x7ff3640b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 10 monclient(hunting): picked mon.noname-c con 0x7ff36414be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff364142710 con 0x7ff3640b1730 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff364142410 con 0x7ff3640b4d60 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff364140d00 con 0x7ff36414be90 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.343+0000 7ff36920f640 10 monclient(hunting): authenticate will time out at 5468.612596s 2024-03-25T19:50:56.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.349+0000 7ff36359e640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7ff36414be90 0x7ff3641e2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:48740/0 (socket says 172.21.15.69:48740) 2024-03-25T19:50:56.349 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.349+0000 7ff36359e640 1 -- 172.21.15.69:0/1421432829 learned_addr learned my addr 172.21.15.69:0/1421432829 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:50:56.350 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.350+0000 7ff34b7fe640 10 client.?.objecter ms_handle_connect 0x7ff36414be90 2024-03-25T19:50:56.350 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.350+0000 7ff34b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.361+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750669175 0 0) 0x7ff364140d00 con 0x7ff36414be90 2024-03-25T19:50:56.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.361+0000 7ff34b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.361+0000 7ff34b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.361 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.361+0000 7ff34b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.361+0000 7ff34b7fe640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:50:56.362 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.361+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff350004eb0 con 0x7ff36414be90 2024-03-25T19:50:56.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.365+0000 7ff34b7fe640 10 client.?.objecter ms_handle_connect 0x7ff3640b4d60 2024-03-25T19:50:56.365 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.365+0000 7ff34b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.368+0000 7ff34b7fe640 10 client.?.objecter ms_handle_connect 0x7ff3640b1730 2024-03-25T19:50:56.368 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.368+0000 7ff34b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.375+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512458535 0 0) 0x7ff364142410 con 0x7ff3640b4d60 2024-03-25T19:50:56.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.375+0000 7ff34b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.375+0000 7ff34b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.375+0000 7ff34b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.375+0000 7ff34b7fe640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:50:56.375 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.375+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff364140d00 con 0x7ff3640b4d60 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.376+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4045013465 0 0) 0x7ff350004eb0 con 0x7ff36414be90 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.376+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff364142410 con 0x7ff36414be90 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.376+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff35c001ec0 con 0x7ff36414be90 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.376+0000 7ff34b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.376+0000 7ff34b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.376+0000 7ff34b7fe640 10 monclient(hunting): dump: 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:56.376 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:56.377 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:56.377 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.377 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:56.377 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:56.377 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:56.377 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:56.377 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:56.377 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.376+0000 7ff34b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:56.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.388+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981103765 0 0) 0x7ff364142710 con 0x7ff3640b1730 2024-03-25T19:50:56.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.388+0000 7ff34b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.388+0000 7ff34b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.389+0000 7ff34b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.389+0000 7ff34b7fe640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:50:56.389 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.389+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff350004eb0 con 0x7ff3640b1730 2024-03-25T19:50:56.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3828641960 0 0) 0x7ff364142410 con 0x7ff36414be90 2024-03-25T19:50:56.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 >> v1:172.21.15.169:6789/0 conn(0x7ff3640b4d60 legacy=0x7ff364142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 >> v1:172.21.15.69:6789/0 conn(0x7ff3640b1730 legacy=0x7ff3641405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:56.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 1 monclient: found mon.c 2024-03-25T19:50:56.391 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff364139000 con 0x7ff36414be90 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff34b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 5 monclient: authenticate success, global_id 4661 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 10 monclient: _renew_subs 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff364141c30 con 0x7ff36414be90 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 10 monclient: _renew_subs 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3641432b0 con 0x7ff36414be90 2024-03-25T19:50:56.392 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.391+0000 7ff36920f640 1 librados: init done 2024-03-25T19:50:56.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.392+0000 7ff36920f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.392+0000 7ff36920f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.392+0000 7ff36920f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:56.393 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.392+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3641428c0 con 0x7ff36414be90 2024-03-25T19:50:56.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.684+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff35c004e10 con 0x7ff36414be90 2024-03-25T19:50:56.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.684+0000 7ff34b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:56.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.684+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff35c004f80 con 0x7ff36414be90 2024-03-25T19:50:56.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.684+0000 7ff34b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.684+0000 7ff34b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:56.684 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.684+0000 7ff34b7fe640 10 monclient: dump: 2024-03-25T19:50:56.684 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:50:56.684 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.684+0000 7ff36159a640 4 set_mon_vals no callback set 2024-03-25T19:50:56.685 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.685+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7ff35c01af50 con 0x7ff36414be90 2024-03-25T19:50:56.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.685+0000 7ff362d9d640 10 monclient: get_auth_request con 0x7ff35006de20 auth_method 0 2024-03-25T19:50:56.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.686+0000 7ff34b7fe640 10 client.4661.objecter ms_handle_connect 0x7ff35006de20 2024-03-25T19:50:56.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.690+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7ff35c0548e0 con 0x7ff36414be90 2024-03-25T19:50:56.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.690+0000 7ff34b7fe640 10 client.4661.objecter ms_dispatch 0x7ff364000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:50:56.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.690+0000 7ff34b7fe640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:50:56.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.690+0000 7ff34b7fe640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:50:56.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.690+0000 7ff34b7fe640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-25T19:50:56.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.690+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff35c0503f0 con 0x7ff36414be90 2024-03-25T19:50:56.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.690+0000 7ff34b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.691 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.690+0000 7ff34b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:56.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.904+0000 7ff36920f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.904+0000 7ff36920f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.904+0000 7ff36920f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:50:56.904 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.904+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3640627b0 con 0x7ff36414be90 2024-03-25T19:50:56.940 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.939+0000 7ff34b7fe640 1 -- 172.21.15.69:0/1421432829 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7ff35c052320 con 0x7ff36414be90 2024-03-25T19:50:56.940 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.940+0000 7ff34b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.940 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.940+0000 7ff34b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:56.940 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:50:56.940 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.942+0000 7ff36920f640 10 librados: watch_flush enter 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.942+0000 7ff36920f640 10 librados: watch_flush exit 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.942+0000 7ff36920f640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.942+0000 7ff36920f640 10 client.4661.objecter successfully canceled tick 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.942+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 >> v1:172.21.15.169:6800/3072548259 conn(0x7ff35006de20 legacy=0x7ff3500702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.942+0000 7ff36920f640 10 monclient: shutdown 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.942+0000 7ff36920f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.942+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 >> v1:172.21.15.69:6790/0 conn(0x7ff36414be90 legacy=0x7ff3641e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.943+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 shutdown_connections 2024-03-25T19:50:56.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.943+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 >> 172.21.15.69:0/1421432829 conn(0x7ff3640617a0 msgr2=0x7ff3640b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:56.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.943+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 shutdown_connections 2024-03-25T19:50:56.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.943+0000 7ff36920f640 1 -- 172.21.15.69:0/1421432829 wait complete. 2024-03-25T19:50:56.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:50:56.943+0000 7ff36920f640 1 librados: shutdown 2024-03-25T19:50:56.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:03.762 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:03.874 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.874+0000 7f3b5e8ae640 10 monclient: get_monmap_and_config 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.874+0000 7f3b5e8ae640 10 monclient: build_initial_monmap 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.874+0000 7f3b5e8ae640 10 monclient: monmap: 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:51:03.876097+0000 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:51:03.876097+0000 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:51:03.875 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.875+0000 7f3b5e8ae640 1 Processor -- start 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.875+0000 7f3b5e8ae640 1 -- start start 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.875+0000 7f3b5e8ae640 10 monclient: init 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 10 monclient(hunting): picked mon.noname-c con 0x7f3b580b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3b580b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3b5814be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580af940 con 0x7f3b580b1730 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580b0a70 con 0x7f3b580b4d60 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580b0d50 con 0x7f3b5814be90 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:03.877 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.876+0000 7f3b5e8ae640 10 monclient(hunting): authenticate will time out at 5476.145029s 2024-03-25T19:51:03.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.878+0000 7f3b57fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f3b580b4d60 0x7f3b5813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:57320/0 (socket says 172.21.15.69:57320) 2024-03-25T19:51:03.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.878+0000 7f3b57fff640 1 -- 172.21.15.69:0/276934524 learned_addr learned my addr 172.21.15.69:0/276934524 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:03.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.896+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866417397 0 0) 0x7f3b580b0a70 con 0x7f3b580b4d60 2024-03-25T19:51:03.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.896+0000 7f3b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.896+0000 7f3b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.896+0000 7f3b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.896+0000 7f3b56ffd640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:51:03.896 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.896+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b40003520 con 0x7f3b580b4d60 2024-03-25T19:51:03.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.901+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245766892 0 0) 0x7f3b580af940 con 0x7f3b580b1730 2024-03-25T19:51:03.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.901+0000 7f3b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.901+0000 7f3b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.901+0000 7f3b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.901+0000 7f3b56ffd640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:51:03.901 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.901+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b580b0a70 con 0x7f3b580b1730 2024-03-25T19:51:03.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3501343684 0 0) 0x7f3b40003520 con 0x7f3b580b4d60 2024-03-25T19:51:03.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 >> v1:172.21.15.69:6790/0 conn(0x7f3b5814be90 legacy=0x7f3b5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 >> v1:172.21.15.69:6789/0 conn(0x7f3b580b1730 legacy=0x7f3b580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:03.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 1 monclient: found mon.noname-c 2024-03-25T19:51:03.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:03.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:51:03.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b580b1030 con 0x7f3b580b4d60 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3b48001690 con 0x7f3b580b4d60 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b56ffd640 10 monclient: dump: 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:03.919 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:03.920 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:03.920 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:03.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b5e8ae640 5 monclient: authenticate success, global_id 4666 2024-03-25T19:51:03.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.918+0000 7f3b5e8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:03.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.938+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b480017f0 con 0x7f3b580b4d60 2024-03-25T19:51:03.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.938+0000 7f3b56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:03.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.938+0000 7f3b56ffd640 1 -- 172.21.15.69:0/276934524 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3b48001c00 con 0x7f3b580b4d60 2024-03-25T19:51:03.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.938+0000 7f3b5e8ae640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:03.938 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.938+0000 7f3b5e8ae640 4 set_mon_vals no callback set 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.938+0000 7f3b56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.938+0000 7f3b56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.938+0000 7f3b56ffd640 10 monclient: dump: 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:03.939 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:03.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.939+0000 7f3b5e8ae640 10 monclient: shutdown 2024-03-25T19:51:03.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.939+0000 7f3b5e8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:03.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.939+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/276934524 >> v1:172.21.15.169:6789/0 conn(0x7f3b580b4d60 legacy=0x7f3b5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.939+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/276934524 shutdown_connections 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.939+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/276934524 >> 172.21.15.69:0/276934524 conn(0x7f3b580617a0 msgr2=0x7f3b58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.939+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/276934524 shutdown_connections 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.939+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/276934524 wait complete. 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 10 monclient: build_initial_monmap 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 10 monclient: monmap: 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:51:03.942 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 1 librados: starting msgr at 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 1 librados: starting objecter 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 1 Processor -- start 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 1 -- start start 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 1 librados: setting wanted keys 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 1 librados: calling monclient init 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.940+0000 7f3b5e8ae640 10 monclient: init 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3b580b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 10 monclient(hunting): picked mon.noname-c con 0x7f3b580b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3b5814be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b581d8750 con 0x7f3b5814be90 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b58140a90 con 0x7f3b580b1730 2024-03-25T19:51:03.943 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b58140c20 con 0x7f3b580b4d60 2024-03-25T19:51:03.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:03.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.941+0000 7f3b5e8ae640 10 monclient(hunting): authenticate will time out at 5476.210190s 2024-03-25T19:51:03.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.953+0000 7f3b5ce4b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3b5814be90 0x7f3b581e21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36934/0 (socket says 172.21.15.69:36934) 2024-03-25T19:51:03.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.953+0000 7f3b5ce4b640 1 -- 172.21.15.69:0/3699550651 learned_addr learned my addr 172.21.15.69:0/3699550651 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:03.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.954+0000 7f3b54ff9640 10 client.?.objecter ms_handle_connect 0x7f3b5814be90 2024-03-25T19:51:03.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.954+0000 7f3b54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.961+0000 7f3b54ff9640 10 client.?.objecter ms_handle_connect 0x7f3b580b4d60 2024-03-25T19:51:03.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.961+0000 7f3b54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.966+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025616508 0 0) 0x7f3b581d8750 con 0x7f3b5814be90 2024-03-25T19:51:03.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.966+0000 7f3b54ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.966+0000 7f3b54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.966+0000 7f3b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.966+0000 7f3b54ff9640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:51:03.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.966+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b44004eb0 con 0x7f3b5814be90 2024-03-25T19:51:03.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.966+0000 7f3b54ff9640 10 client.?.objecter ms_handle_connect 0x7f3b580b1730 2024-03-25T19:51:03.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.966+0000 7f3b54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.968 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.968+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498941595 0 0) 0x7f3b58140c20 con 0x7f3b580b4d60 2024-03-25T19:51:03.968 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.968+0000 7f3b54ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.968 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.968+0000 7f3b54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.968 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.968+0000 7f3b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.968 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.968+0000 7f3b54ff9640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:51:03.968 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.968+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b581d8750 con 0x7f3b580b4d60 2024-03-25T19:51:03.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.981+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147163085 0 0) 0x7f3b58140a90 con 0x7f3b580b1730 2024-03-25T19:51:03.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.981+0000 7f3b54ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.981+0000 7f3b54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.981+0000 7f3b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.981+0000 7f3b54ff9640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:51:03.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.981+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b58140c20 con 0x7f3b580b1730 2024-03-25T19:51:03.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.985+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2409475350 0 0) 0x7f3b44004eb0 con 0x7f3b5814be90 2024-03-25T19:51:03.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.985+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b58140a90 con 0x7f3b5814be90 2024-03-25T19:51:03.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.985+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3b50001c90 con 0x7f3b5814be90 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.985+0000 7f3b54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.985+0000 7f3b54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.985+0000 7f3b54ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:03.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.985+0000 7f3b54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.988+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 275923075 0 0) 0x7f3b581d8750 con 0x7f3b580b4d60 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.988+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b44004eb0 con 0x7f3b580b4d60 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.988+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3b4c0018b0 con 0x7f3b580b4d60 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.988+0000 7f3b54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.988+0000 7f3b54ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.988+0000 7f3b54ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:03.989 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:03.990 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:03.990 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:03.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2056755974 0 0) 0x7f3b58140a90 con 0x7f3b5814be90 2024-03-25T19:51:03.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 >> v1:172.21.15.69:6790/0 conn(0x7f3b580b4d60 legacy=0x7f3b58140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 >> v1:172.21.15.169:6789/0 conn(0x7f3b580b1730 legacy=0x7f3b581d8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:03.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 1 monclient: found mon.a 2024-03-25T19:51:03.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:03.993 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5813b900 con 0x7f3b5814be90 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.992+0000 7f3b5e8ae640 5 monclient: authenticate success, global_id 4698 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.993+0000 7f3b5e8ae640 10 monclient: _renew_subs 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.993+0000 7f3b5e8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.993+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5813bb70 con 0x7f3b5814be90 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.993+0000 7f3b5e8ae640 10 client.4698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.993+0000 7f3b5e8ae640 10 monclient: _renew_subs 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.993+0000 7f3b5e8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.993+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b581e3aa0 con 0x7f3b5814be90 2024-03-25T19:51:03.994 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.993+0000 7f3b5e8ae640 1 librados: init done 2024-03-25T19:51:03.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.994+0000 7f3b5e8ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:03.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.994+0000 7f3b5e8ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:03.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.994+0000 7f3b5e8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:51:03.995 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:03.994+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b581d7e90 con 0x7f3b5814be90 2024-03-25T19:51:04.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.297+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b50003990 con 0x7f3b5814be90 2024-03-25T19:51:04.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.297+0000 7f3b54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:04.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.297+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f3b50003d60 con 0x7f3b5814be90 2024-03-25T19:51:04.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.297+0000 7f3b54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:04.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.297+0000 7f3b54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:04.297 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.297+0000 7f3b54ff9640 10 monclient: dump: 2024-03-25T19:51:04.297 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:04.298 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.297+0000 7f3b567fc640 4 set_mon_vals no callback set 2024-03-25T19:51:04.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.298+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f3b50003f80 con 0x7f3b5814be90 2024-03-25T19:51:04.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.298+0000 7f3b57fff640 10 monclient: get_auth_request con 0x7f3b4406df70 auth_method 0 2024-03-25T19:51:04.299 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.299+0000 7f3b54ff9640 10 client.4698.objecter ms_handle_connect 0x7f3b4406df70 2024-03-25T19:51:04.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.303+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7f3b50054a50 con 0x7f3b5814be90 2024-03-25T19:51:04.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.303+0000 7f3b54ff9640 10 client.4698.objecter ms_dispatch 0x7f3b58000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:04.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.303+0000 7f3b54ff9640 3 client.4698.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:04.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.303+0000 7f3b54ff9640 3 client.4698.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:04.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.303+0000 7f3b54ff9640 20 client.4698.objecter dump_active .. 0 homeless 2024-03-25T19:51:04.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.303+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b50050560 con 0x7f3b5814be90 2024-03-25T19:51:04.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.303+0000 7f3b54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:04.304 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.303+0000 7f3b54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:04.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.558+0000 7f3b5e8ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.558+0000 7f3b5e8ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.558+0000 7f3b5e8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:51:04.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.558+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b581d7910 con 0x7f3b5814be90 2024-03-25T19:51:04.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.580+0000 7f3b54ff9640 1 -- 172.21.15.69:0/3699550651 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7f3b50052490 con 0x7f3b5814be90 2024-03-25T19:51:04.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.580+0000 7f3b54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.581 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.580+0000 7f3b54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:04.581 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:51:04.581 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:04.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 10 librados: watch_flush enter 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 10 librados: watch_flush exit 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 20 client.4698.objecter shutdown clearing up homeless session... 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 10 client.4698.objecter successfully canceled tick 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 >> v1:172.21.15.169:6800/3072548259 conn(0x7f3b4406df70 legacy=0x7f3b44070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 10 monclient: shutdown 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 >> v1:172.21.15.69:6789/0 conn(0x7f3b5814be90 legacy=0x7f3b581e21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 shutdown_connections 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 >> 172.21.15.69:0/3699550651 conn(0x7f3b580617a0 msgr2=0x7f3b5813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 shutdown_connections 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 1 -- 172.21.15.69:0/3699550651 wait complete. 2024-03-25T19:51:04.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:04.583+0000 7f3b5e8ae640 1 librados: shutdown 2024-03-25T19:51:04.595 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:11.402 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:11.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.516+0000 7ff9ff983640 10 monclient: get_monmap_and_config 2024-03-25T19:51:11.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.516+0000 7ff9ff983640 10 monclient: build_initial_monmap 2024-03-25T19:51:11.516 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.516+0000 7ff9ff983640 10 monclient: monmap: 2024-03-25T19:51:11.516 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:11.516 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:11.516 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:51:11.517660+0000 2024-03-25T19:51:11.516 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:51:11.517660+0000 2024-03-25T19:51:11.516 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:11.517 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:11.517 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:11.517 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:51:11.517 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:51:11.517 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:11.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.517+0000 7ff9ff983640 1 Processor -- start 2024-03-25T19:51:11.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.517+0000 7ff9ff983640 1 -- start start 2024-03-25T19:51:11.517 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.517+0000 7ff9ff983640 10 monclient: init 2024-03-25T19:51:11.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:11.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:11.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f80a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:11.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:11.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:11.518 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f8142520 con 0x7ff9f813f710 2024-03-25T19:51:11.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f80b28a0 con 0x7ff9f81430c0 2024-03-25T19:51:11.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f80b2a30 con 0x7ff9f80a98f0 2024-03-25T19:51:11.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:11.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.518+0000 7ff9ff983640 10 monclient(hunting): authenticate will time out at 5483.787178s 2024-03-25T19:51:11.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.532+0000 7ff9fd71f640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7ff9f80a98f0 0x7ff9f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:48020/0 (socket says 172.21.15.69:48020) 2024-03-25T19:51:11.532 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.532+0000 7ff9fd71f640 1 -- 172.21.15.69:0/364090530 learned_addr learned my addr 172.21.15.69:0/364090530 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:11.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.540+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787866653 0 0) 0x7ff9f80b2a30 con 0x7ff9f80a98f0 2024-03-25T19:51:11.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.540+0000 7ff9effff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.540+0000 7ff9effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.540+0000 7ff9effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.540+0000 7ff9effff640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:51:11.541 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.540+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e4003520 con 0x7ff9f80a98f0 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2001545268 0 0) 0x7ff9e4003520 con 0x7ff9f80a98f0 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 >> v1:172.21.15.169:6789/0 conn(0x7ff9f81430c0 legacy=0x7ff9f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 >> v1:172.21.15.69:6789/0 conn(0x7ff9f813f710 legacy=0x7ff9f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 1 monclient: found mon.noname-b 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f80b44c0 con 0x7ff9f80a98f0 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 10 monclient: _finish_auth 0 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff9f4001690 con 0x7ff9f80a98f0 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9ff983640 5 monclient: authenticate success, global_id 4670 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9ff983640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:11.549 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9effff640 10 monclient: dump: 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:11.550 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.548+0000 7ff9ff983640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:11.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.564+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9f40017f0 con 0x7ff9f80a98f0 2024-03-25T19:51:11.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.564+0000 7ff9effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:11.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.564+0000 7ff9effff640 1 -- 172.21.15.69:0/364090530 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff9f4001c00 con 0x7ff9f80a98f0 2024-03-25T19:51:11.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.564+0000 7ff9ff983640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:11.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.564+0000 7ff9ff983640 4 set_mon_vals no callback set 2024-03-25T19:51:11.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.564+0000 7ff9effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.564 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.564+0000 7ff9effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.564+0000 7ff9effff640 10 monclient: dump: 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:11.565 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.565+0000 7ff9ff983640 10 monclient: shutdown 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.565+0000 7ff9ff983640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.565+0000 7ff9ff983640 1 -- 172.21.15.69:0/364090530 >> v1:172.21.15.69:6790/0 conn(0x7ff9f80a98f0 legacy=0x7ff9f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.565+0000 7ff9ff983640 1 -- 172.21.15.69:0/364090530 shutdown_connections 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.565+0000 7ff9ff983640 1 -- 172.21.15.69:0/364090530 >> 172.21.15.69:0/364090530 conn(0x7ff9f80b7ea0 msgr2=0x7ff9f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.565+0000 7ff9ff983640 1 -- 172.21.15.69:0/364090530 shutdown_connections 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.565+0000 7ff9ff983640 1 -- 172.21.15.69:0/364090530 wait complete. 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.566+0000 7ff9ff983640 10 monclient: build_initial_monmap 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.566+0000 7ff9ff983640 10 monclient: monmap: 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:51:11.567 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.566+0000 7ff9ff983640 1 librados: starting msgr at 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.566+0000 7ff9ff983640 1 librados: starting objecter 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.566+0000 7ff9ff983640 1 Processor -- start 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 1 -- start start 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 1 librados: setting wanted keys 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 1 librados: calling monclient init 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 10 monclient: init 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f80a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f81430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f81dac50 con 0x7ff9f813f710 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f80b16c0 con 0x7ff9f80a98f0 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f8136b70 con 0x7ff9f81430c0 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.567+0000 7ff9ff983640 10 monclient(hunting): authenticate will time out at 5483.836340s 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.570+0000 7ff9fcf1e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff9f813f710 0x7ff9f81db0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54594/0 (socket says 172.21.15.69:54594) 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.570+0000 7ff9fcf1e640 1 -- 172.21.15.69:0/337347389 learned_addr learned my addr 172.21.15.69:0/337347389 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.576+0000 7ff9edffb640 10 client.?.objecter ms_handle_connect 0x7ff9f813f710 2024-03-25T19:51:11.576 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.576+0000 7ff9edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.580+0000 7ff9edffb640 10 client.?.objecter ms_handle_connect 0x7ff9f80a98f0 2024-03-25T19:51:11.580 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.580+0000 7ff9edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.583 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.584+0000 7ff9edffb640 10 client.?.objecter ms_handle_connect 0x7ff9f81430c0 2024-03-25T19:51:11.584 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.584+0000 7ff9edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.589+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433713772 0 0) 0x7ff9f81dac50 con 0x7ff9f813f710 2024-03-25T19:51:11.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.589+0000 7ff9edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.589+0000 7ff9edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.589+0000 7ff9edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.589+0000 7ff9edffb640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:51:11.589 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.589+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc004eb0 con 0x7ff9f813f710 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.593+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199551335 0 0) 0x7ff9f8136b70 con 0x7ff9f81430c0 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.593+0000 7ff9edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.593+0000 7ff9edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.593+0000 7ff9edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.593+0000 7ff9edffb640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.593+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f81dac50 con 0x7ff9f81430c0 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.603+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958958772 0 0) 0x7ff9f80b16c0 con 0x7ff9f80a98f0 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.603+0000 7ff9edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.603+0000 7ff9edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.603+0000 7ff9edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.603+0000 7ff9edffb640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.603+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f8136b70 con 0x7ff9f80a98f0 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.608+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2585991058 0 0) 0x7ff9dc004eb0 con 0x7ff9f813f710 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.608+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9f80b16c0 con 0x7ff9f813f710 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.608+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff9e80019d0 con 0x7ff9f813f710 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.608+0000 7ff9edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.608+0000 7ff9edffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.608+0000 7ff9edffb640 10 monclient(hunting): dump: 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.608+0000 7ff9edffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:11.613 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.613+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 888933391 0 0) 0x7ff9f81dac50 con 0x7ff9f81430c0 2024-03-25T19:51:11.613 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.613+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9dc004eb0 con 0x7ff9f81430c0 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.616+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff9f0001bb0 con 0x7ff9f81430c0 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.616+0000 7ff9edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.616+0000 7ff9edffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.616+0000 7ff9edffb640 10 monclient(hunting): dump: 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.616 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:11.617 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:11.617 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:11.617 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:11.617 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:11.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3036822013 0 0) 0x7ff9dc004eb0 con 0x7ff9f81430c0 2024-03-25T19:51:11.629 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 >> v1:172.21.15.169:6789/0 conn(0x7ff9f80a98f0 legacy=0x7ff9f81375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 >> v1:172.21.15.69:6789/0 conn(0x7ff9f813f710 legacy=0x7ff9f81db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 1 monclient: found mon.c 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f80b11a0 con 0x7ff9f81430c0 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 5 monclient: authenticate success, global_id 4673 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 10 monclient: _renew_subs 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f80b13f0 con 0x7ff9f81430c0 2024-03-25T19:51:11.630 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:11.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 10 monclient: _renew_subs 2024-03-25T19:51:11.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:11.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f8137ea0 con 0x7ff9f81430c0 2024-03-25T19:51:11.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.629+0000 7ff9ff983640 1 librados: init done 2024-03-25T19:51:11.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.630+0000 7ff9ff983640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.630+0000 7ff9ff983640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.630+0000 7ff9ff983640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:11.631 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.630+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f80627b0 con 0x7ff9f81430c0 2024-03-25T19:51:11.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.919+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9f0002080 con 0x7ff9f81430c0 2024-03-25T19:51:11.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.919+0000 7ff9edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:11.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.920+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff9f0003b30 con 0x7ff9f81430c0 2024-03-25T19:51:11.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.920+0000 7ff9edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.920+0000 7ff9edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:11.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.920+0000 7ff9edffb640 10 monclient: dump: 2024-03-25T19:51:11.920 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:11.920 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:11.921 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.920+0000 7ff9ef7fe640 4 set_mon_vals no callback set 2024-03-25T19:51:11.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.920+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7ff9f0003d50 con 0x7ff9f81430c0 2024-03-25T19:51:11.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.921+0000 7ff9fd71f640 10 monclient: get_auth_request con 0x7ff9dc06df20 auth_method 0 2024-03-25T19:51:11.922 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.921+0000 7ff9edffb640 10 client.4673.objecter ms_handle_connect 0x7ff9dc06df20 2024-03-25T19:51:11.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.926+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7ff9f0053880 con 0x7ff9f81430c0 2024-03-25T19:51:11.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.926+0000 7ff9edffb640 10 client.4673.objecter ms_dispatch 0x7ff9f8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:11.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.926+0000 7ff9edffb640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:11.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.926+0000 7ff9edffb640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:11.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.926+0000 7ff9edffb640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-25T19:51:11.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.926+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9f004f1c0 con 0x7ff9f81430c0 2024-03-25T19:51:11.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.926+0000 7ff9edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:11.926+0000 7ff9edffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:12.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.165+0000 7ff9ff983640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.165+0000 7ff9ff983640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.165 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.165+0000 7ff9ff983640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:12.166 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.165+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9f813e250 con 0x7ff9f81430c0 2024-03-25T19:51:12.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.186+0000 7ff9edffb640 1 -- 172.21.15.69:0/337347389 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7ff9f00512c0 con 0x7ff9f81430c0 2024-03-25T19:51:12.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.186+0000 7ff9edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.186+0000 7ff9edffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:12.186 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:51:12.186 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 10 librados: watch_flush enter 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 10 librados: watch_flush exit 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 10 client.4673.objecter successfully canceled tick 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 >> v1:172.21.15.169:6800/3072548259 conn(0x7ff9dc06df20 legacy=0x7ff9dc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 10 monclient: shutdown 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 >> v1:172.21.15.69:6790/0 conn(0x7ff9f81430c0 legacy=0x7ff9f81e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 shutdown_connections 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 >> 172.21.15.69:0/337347389 conn(0x7ff9f80b7ea0 msgr2=0x7ff9f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:12.189 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 shutdown_connections 2024-03-25T19:51:12.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 1 -- 172.21.15.69:0/337347389 wait complete. 2024-03-25T19:51:12.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:12.188+0000 7ff9ff983640 1 librados: shutdown 2024-03-25T19:51:12.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:13.880 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:51.844 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:13.888 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:51.852 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:13.923 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:51.887 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:13.945 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:51.900 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:13.949 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:51.903 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:13.961 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:51.925 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:13.969 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:51.933 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:13.975 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:51.929 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:14.000 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:51.964 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.006 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.122+0000 7f16b69a8640 10 monclient: get_monmap_and_config 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.123+0000 7f16b69a8640 10 monclient: build_initial_monmap 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.123+0000 7f16b69a8640 10 monclient: monmap: 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:51:19.124236+0000 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:51:19.124236+0000 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:19.124 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:19.124 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:19.124 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:51:19.124 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:51:19.124 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.123+0000 7f16b69a8640 1 Processor -- start 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 1 -- start start 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 10 monclient: init 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 10 monclient(hunting): picked mon.noname-a con 0x7f16b00a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 10 monclient(hunting): picked mon.noname-b con 0x7f16b013f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 10 monclient(hunting): picked mon.noname-c con 0x7f16b01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b0142520 con 0x7f16b00a98f0 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b00b28a0 con 0x7f16b01430c0 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b00b2a30 con 0x7f16b013f710 2024-03-25T19:51:19.125 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:19.126 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.124+0000 7f16b69a8640 10 monclient(hunting): authenticate will time out at 5491.393419s 2024-03-25T19:51:19.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.130+0000 7f16af7fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f16b013f710 0x7f16b01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:52816/0 (socket says 172.21.15.69:52816) 2024-03-25T19:51:19.130 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.130+0000 7f16af7fe640 1 -- 172.21.15.69:0/4001551105 learned_addr learned my addr 172.21.15.69:0/4001551105 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.139+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539799493 0 0) 0x7f16b00b2a30 con 0x7f16b013f710 2024-03-25T19:51:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.139+0000 7f16aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.139+0000 7f16aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.139+0000 7f16aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.139+0000 7f16aeffd640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:51:19.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.139+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1698003520 con 0x7f16b013f710 2024-03-25T19:51:19.143 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.143+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180015929 0 0) 0x7f16b00b28a0 con 0x7f16b01430c0 2024-03-25T19:51:19.143 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.143+0000 7f16aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.143 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.143+0000 7f16aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.143 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.143+0000 7f16aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.143 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.143+0000 7f16aeffd640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:51:19.143 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.143+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b00b2a30 con 0x7f16b01430c0 2024-03-25T19:51:19.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2210386784 0 0) 0x7f1698003520 con 0x7f16b013f710 2024-03-25T19:51:19.157 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 >> v1:172.21.15.169:6789/0 conn(0x7f16b01430c0 legacy=0x7f16b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 >> v1:172.21.15.69:6789/0 conn(0x7f16b00a98f0 legacy=0x7f16b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 1 monclient: found mon.noname-b 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b00b44c0 con 0x7f16b013f710 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f16a4001690 con 0x7f16b013f710 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16aeffd640 10 monclient: dump: 2024-03-25T19:51:19.158 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16b69a8640 5 monclient: authenticate success, global_id 4676 2024-03-25T19:51:19.159 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.157+0000 7f16b69a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.174+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16a40017f0 con 0x7f16b013f710 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.174+0000 7f16aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.174+0000 7f16aeffd640 1 -- 172.21.15.69:0/4001551105 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f16a4001c00 con 0x7f16b013f710 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.174+0000 7f16b69a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.174+0000 7f16b69a8640 4 set_mon_vals no callback set 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.174+0000 7f16aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.174+0000 7f16aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.174+0000 7f16aeffd640 10 monclient: dump: 2024-03-25T19:51:19.174 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:19.175 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:19.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.175+0000 7f16b69a8640 10 monclient: shutdown 2024-03-25T19:51:19.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.175+0000 7f16b69a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:19.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.175+0000 7f16b69a8640 1 -- 172.21.15.69:0/4001551105 >> v1:172.21.15.69:6790/0 conn(0x7f16b013f710 legacy=0x7f16b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.175+0000 7f16b69a8640 1 -- 172.21.15.69:0/4001551105 shutdown_connections 2024-03-25T19:51:19.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.175+0000 7f16b69a8640 1 -- 172.21.15.69:0/4001551105 >> 172.21.15.69:0/4001551105 conn(0x7f16b00b7ea0 msgr2=0x7f16b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:19.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.175+0000 7f16b69a8640 1 -- 172.21.15.69:0/4001551105 shutdown_connections 2024-03-25T19:51:19.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.175+0000 7f16b69a8640 1 -- 172.21.15.69:0/4001551105 wait complete. 2024-03-25T19:51:19.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 10 monclient: build_initial_monmap 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 10 monclient: monmap: 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 1 librados: starting msgr at 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 1 librados: starting objecter 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 1 Processor -- start 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 1 -- start start 2024-03-25T19:51:19.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 1 librados: setting wanted keys 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 1 librados: calling monclient init 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.176+0000 7f16b69a8640 10 monclient: init 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 10 monclient(hunting): picked mon.noname-b con 0x7f16b00a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 10 monclient(hunting): picked mon.noname-a con 0x7f16b013f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 10 monclient(hunting): picked mon.noname-c con 0x7f16b01430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b0138a50 con 0x7f16b013f710 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b00ab9e0 con 0x7f16b00a98f0 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b01d7880 con 0x7f16b01430c0 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:19.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.177+0000 7f16b69a8640 10 monclient(hunting): authenticate will time out at 5491.445790s 2024-03-25T19:51:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.180+0000 7f16b4f45640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f16b01430c0 0x7f16b01e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:52850/0 (socket says 172.21.15.69:52850) 2024-03-25T19:51:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.180+0000 7f16b4f45640 1 -- 172.21.15.69:0/3962305813 learned_addr learned my addr 172.21.15.69:0/3962305813 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.181+0000 7f16acff9640 10 client.?.objecter ms_handle_connect 0x7f16b01430c0 2024-03-25T19:51:19.181 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.181+0000 7f16acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.189+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733867374 0 0) 0x7f16b01d7880 con 0x7f16b01430c0 2024-03-25T19:51:19.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.190+0000 7f16acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.190+0000 7f16acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.190+0000 7f16acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.190+0000 7f16acff9640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:51:19.190 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.190+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f169c004eb0 con 0x7f16b01430c0 2024-03-25T19:51:19.194 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.194+0000 7f16acff9640 10 client.?.objecter ms_handle_connect 0x7f16b00a98f0 2024-03-25T19:51:19.194 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.194+0000 7f16acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.196+0000 7f16acff9640 10 client.?.objecter ms_handle_connect 0x7f16b013f710 2024-03-25T19:51:19.196 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.196+0000 7f16acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.202+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2169541239 0 0) 0x7f169c004eb0 con 0x7f16b01430c0 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.202+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16b01d7880 con 0x7f16b01430c0 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.202+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f16a8001c70 con 0x7f16b01430c0 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.202+0000 7f16acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.202+0000 7f16acff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.202+0000 7f16acff9640 10 monclient(hunting): dump: 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:19.202 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:19.203 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:19.203 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.203 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:19.203 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:19.203 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:19.203 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:19.203 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:19.203 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.202+0000 7f16acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:19.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.205+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975806039 0 0) 0x7f16b0138a50 con 0x7f16b013f710 2024-03-25T19:51:19.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.205+0000 7f16acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.205+0000 7f16acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.205+0000 7f16acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.205+0000 7f16acff9640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:51:19.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.205+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f169c004eb0 con 0x7f16b013f710 2024-03-25T19:51:19.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.206+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266994358 0 0) 0x7f16b00ab9e0 con 0x7f16b00a98f0 2024-03-25T19:51:19.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.206+0000 7f16acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.206+0000 7f16acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.206+0000 7f16acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.206+0000 7f16acff9640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:51:19.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.206+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b0138a50 con 0x7f16b00a98f0 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2394821512 0 0) 0x7f16b01d7880 con 0x7f16b01430c0 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 >> v1:172.21.15.169:6789/0 conn(0x7f16b00a98f0 legacy=0x7f16b0138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 >> v1:172.21.15.69:6789/0 conn(0x7f16b013f710 legacy=0x7f16b01d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 1 monclient: found mon.c 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b01d84b0 con 0x7f16b01430c0 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:19.213 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16b69a8640 5 monclient: authenticate success, global_id 4679 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16b69a8640 10 monclient: _renew_subs 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16b69a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.212+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b01d51f0 con 0x7f16b01430c0 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 10 monclient: _renew_subs 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b0138e60 con 0x7f16b01430c0 2024-03-25T19:51:19.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 1 librados: init done 2024-03-25T19:51:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:19.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.213+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b01d76e0 con 0x7f16b01430c0 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.519+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16a8003970 con 0x7f16b01430c0 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.519+0000 7f16acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.519+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f16a8003be0 con 0x7f16b01430c0 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.519+0000 7f16acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.519+0000 7f16acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.519+0000 7f16acff9640 10 monclient: dump: 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.521 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:19.521 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:19.521 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:19.521 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:19.521 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:19.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.519+0000 7f16ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:51:19.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.520+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f16a8003e00 con 0x7f16b01430c0 2024-03-25T19:51:19.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.521+0000 7f16affff640 10 monclient: get_auth_request con 0x7f169c06de80 auth_method 0 2024-03-25T19:51:19.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.521+0000 7f16acff9640 10 client.4679.objecter ms_handle_connect 0x7f169c06de80 2024-03-25T19:51:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.525+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7f16a8054960 con 0x7f16b01430c0 2024-03-25T19:51:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.525+0000 7f16acff9640 10 client.4679.objecter ms_dispatch 0x7f16b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.525+0000 7f16acff9640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.525+0000 7f16acff9640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.526+0000 7f16acff9640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-25T19:51:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.526+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16a8050470 con 0x7f16b01430c0 2024-03-25T19:51:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.526+0000 7f16acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.526 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.526+0000 7f16acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:19.806 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.806+0000 7f16b69a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.807 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.806+0000 7f16b69a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.807 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.806+0000 7f16b69a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:19.807 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.806+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16b00627b0 con 0x7f16b01430c0 2024-03-25T19:51:19.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.838+0000 7f16acff9640 1 -- 172.21.15.69:0/3962305813 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7f16a8004800 con 0x7f16b01430c0 2024-03-25T19:51:19.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.838+0000 7f16acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.839 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.838+0000 7f16acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:19.839 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:51:19.839 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:19.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.840+0000 7f16b69a8640 10 librados: watch_flush enter 2024-03-25T19:51:19.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 10 librados: watch_flush exit 2024-03-25T19:51:19.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-25T19:51:19.841 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 10 client.4679.objecter successfully canceled tick 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 >> v1:172.21.15.169:6800/3072548259 conn(0x7f169c06de80 legacy=0x7f169c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 10 monclient: shutdown 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 >> v1:172.21.15.69:6790/0 conn(0x7f16b01430c0 legacy=0x7f16b01e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 shutdown_connections 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 >> 172.21.15.69:0/3962305813 conn(0x7f16b00b7ea0 msgr2=0x7f16b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 shutdown_connections 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 1 -- 172.21.15.69:0/3962305813 wait complete. 2024-03-25T19:51:19.842 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:19.841+0000 7f16b69a8640 1 librados: shutdown 2024-03-25T19:51:19.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:26.660 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.766+0000 7f5d20590640 10 monclient: get_monmap_and_config 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.766+0000 7f5d20590640 10 monclient: build_initial_monmap 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.766+0000 7f5d20590640 10 monclient: monmap: 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:51:26.767419+0000 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:51:26.767419+0000 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:26.766 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:26.767 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:51:26.767 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:51:26.767 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.766+0000 7f5d20590640 1 Processor -- start 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.766+0000 7f5d20590640 1 -- start start 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.766+0000 7f5d20590640 10 monclient: init 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 10 monclient(hunting): picked mon.noname-c con 0x7f5d180b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 10 monclient(hunting): picked mon.noname-a con 0x7f5d180b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1814be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d180af940 con 0x7f5d180b1730 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d180b0a70 con 0x7f5d180b4d60 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d180b0d50 con 0x7f5d1814be90 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:26.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.767+0000 7f5d20590640 10 monclient(hunting): authenticate will time out at 5499.036262s 2024-03-25T19:51:26.777 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.777+0000 7f5d1e32c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5d180b4d60 0x7f5d1813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:37132/0 (socket says 172.21.15.69:37132) 2024-03-25T19:51:26.778 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.777+0000 7f5d1e32c640 1 -- 172.21.15.69:0/3349251569 learned_addr learned my addr 172.21.15.69:0/3349251569 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:26.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.797+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270347589 0 0) 0x7f5d180b0a70 con 0x7f5d180b4d60 2024-03-25T19:51:26.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.797+0000 7f5d1d32a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.797+0000 7f5d1d32a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.797+0000 7f5d1d32a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.797+0000 7f5d1d32a640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:51:26.797 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.797+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d04003520 con 0x7f5d180b4d60 2024-03-25T19:51:26.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.799+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777502549 0 0) 0x7f5d180af940 con 0x7f5d180b1730 2024-03-25T19:51:26.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.799+0000 7f5d1d32a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.799+0000 7f5d1d32a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.799+0000 7f5d1d32a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.799+0000 7f5d1d32a640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:51:26.799 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.799+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d180b0a70 con 0x7f5d180b1730 2024-03-25T19:51:26.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.800+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688758029 0 0) 0x7f5d180b0d50 con 0x7f5d1814be90 2024-03-25T19:51:26.800 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.800+0000 7f5d1d32a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.800+0000 7f5d1d32a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.800+0000 7f5d1d32a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.800+0000 7f5d1d32a640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:51:26.801 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.800+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d180af940 con 0x7f5d1814be90 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.811+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1930107027 0 0) 0x7f5d180af940 con 0x7f5d1814be90 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.811+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 >> v1:172.21.15.169:6789/0 conn(0x7f5d180b4d60 legacy=0x7f5d1813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 >> v1:172.21.15.69:6789/0 conn(0x7f5d180b1730 legacy=0x7f5d180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 10 monclient: _finish_hunting 0 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 1 monclient: found mon.noname-b 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d180b1030 con 0x7f5d1814be90 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 10 monclient: _finish_auth 0 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 10 monclient: _check_auth_tickets 2024-03-25T19:51:26.812 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5d10001930 con 0x7f5d1814be90 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d20590640 5 monclient: authenticate success, global_id 4682 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d20590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d1d32a640 10 monclient: dump: 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:26.813 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:26.814 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:26.814 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.812+0000 7f5d20590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.822+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d10001a90 con 0x7f5d1814be90 2024-03-25T19:51:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.822+0000 7f5d1d32a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.822+0000 7f5d1d32a640 1 -- 172.21.15.69:0/3349251569 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5d10001ee0 con 0x7f5d1814be90 2024-03-25T19:51:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.822+0000 7f5d20590640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.822+0000 7f5d20590640 4 set_mon_vals no callback set 2024-03-25T19:51:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.822+0000 7f5d1d32a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.822+0000 7f5d1d32a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:26.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.822+0000 7f5d1d32a640 10 monclient: dump: 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:26.823 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:26.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.823+0000 7f5d20590640 10 monclient: shutdown 2024-03-25T19:51:26.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.823+0000 7f5d20590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:26.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.823+0000 7f5d20590640 1 -- 172.21.15.69:0/3349251569 >> v1:172.21.15.69:6790/0 conn(0x7f5d1814be90 legacy=0x7f5d1814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.823+0000 7f5d20590640 1 -- 172.21.15.69:0/3349251569 shutdown_connections 2024-03-25T19:51:26.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.823+0000 7f5d20590640 1 -- 172.21.15.69:0/3349251569 >> 172.21.15.69:0/3349251569 conn(0x7f5d180617a0 msgr2=0x7f5d18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:26.825 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.823+0000 7f5d20590640 1 -- 172.21.15.69:0/3349251569 shutdown_connections 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.823+0000 7f5d20590640 1 -- 172.21.15.69:0/3349251569 wait complete. 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.823+0000 7f5d20590640 10 monclient: build_initial_monmap 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 10 monclient: monmap: 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 1 librados: starting msgr at 2024-03-25T19:51:26.826 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 1 librados: starting objecter 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 1 Processor -- start 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 1 -- start start 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 1 librados: setting wanted keys 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 1 librados: calling monclient init 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 10 monclient: init 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.824+0000 7f5d20590640 10 monclient(hunting): picked mon.noname-c con 0x7f5d180b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.825+0000 7f5d20590640 10 monclient(hunting): picked mon.noname-a con 0x7f5d180b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.825+0000 7f5d20590640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1814be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.825+0000 7f5d20590640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d181409a0 con 0x7f5d180b4d60 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.825+0000 7f5d20590640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d18145060 con 0x7f5d1814be90 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.825+0000 7f5d20590640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d181451f0 con 0x7f5d180b1730 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.825+0000 7f5d20590640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:26.827 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.825+0000 7f5d20590640 10 monclient(hunting): authenticate will time out at 5499.093677s 2024-03-25T19:51:26.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.828+0000 7f5d1e32c640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f5d180b1730 0x7f5d18131d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:42018/0 (socket says 172.21.15.69:42018) 2024-03-25T19:51:26.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.828+0000 7f5d1e32c640 1 -- 172.21.15.69:0/1898470622 learned_addr learned my addr 172.21.15.69:0/1898470622 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:26.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.831+0000 7f5d0effd640 10 client.?.objecter ms_handle_connect 0x7f5d180b1730 2024-03-25T19:51:26.831 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.831+0000 7f5d0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.838+0000 7f5d0effd640 10 client.?.objecter ms_handle_connect 0x7f5d1814be90 2024-03-25T19:51:26.838 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.838+0000 7f5d0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.843+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008235715 0 0) 0x7f5d181451f0 con 0x7f5d180b1730 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.843+0000 7f5d0effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.843+0000 7f5d0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.843+0000 7f5d0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.843+0000 7f5d0effd640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.843+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cfc004eb0 con 0x7f5d180b1730 2024-03-25T19:51:26.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.846+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784861685 0 0) 0x7f5d18145060 con 0x7f5d1814be90 2024-03-25T19:51:26.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.846+0000 7f5d0effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.846+0000 7f5d0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.846+0000 7f5d0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.846+0000 7f5d0effd640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:51:26.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.846+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d181451f0 con 0x7f5d1814be90 2024-03-25T19:51:26.846 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.846+0000 7f5d0effd640 10 client.?.objecter ms_handle_connect 0x7f5d180b4d60 2024-03-25T19:51:26.847 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.846+0000 7f5d0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.851+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4061215200 0 0) 0x7f5cfc004eb0 con 0x7f5d180b1730 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.851+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d18145060 con 0x7f5d180b1730 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.851+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5d14001950 con 0x7f5d180b1730 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.851+0000 7f5d0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.851+0000 7f5d0effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.851+0000 7f5d0effd640 10 monclient(hunting): dump: 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:26.852 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:26.853 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.852+0000 7f5d0effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.855+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1722228569 0 0) 0x7f5d181451f0 con 0x7f5d1814be90 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.855+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cfc004eb0 con 0x7f5d1814be90 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.855+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5d10001da0 con 0x7f5d1814be90 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.855+0000 7f5d0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.855+0000 7f5d0effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.855+0000 7f5d0effd640 10 monclient(hunting): dump: 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3278463251 0 0) 0x7f5d18145060 con 0x7f5d180b1730 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 >> v1:172.21.15.169:6789/0 conn(0x7f5d1814be90 legacy=0x7f5d181d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 >> v1:172.21.15.69:6789/0 conn(0x7f5d180b4d60 legacy=0x7f5d181d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 1 monclient: found mon.c 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d18145380 con 0x7f5d180b1730 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 10 monclient: _finish_auth 0 2024-03-25T19:51:26.858 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 5 monclient: authenticate success, global_id 4685 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 10 monclient: _renew_subs 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d18140380 con 0x7f5d180b1730 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 10 monclient: _renew_subs 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d18132420 con 0x7f5d180b1730 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.856+0000 7f5d20590640 1 librados: init done 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.857+0000 7f5d20590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.857+0000 7f5d20590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.857+0000 7f5d20590640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:26.859 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.857+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d180627b0 con 0x7f5d180b1730 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.888+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d140047e0 con 0x7f5d180b1730 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.888+0000 7f5d0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.888+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5d14004af0 con 0x7f5d180b1730 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.888+0000 7f5d0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.888+0000 7f5d0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.888+0000 7f5d0effd640 10 monclient: dump: 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:26.888 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.889 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:26.889 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:26.889 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:26.889 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:26.889 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:26.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.888+0000 7f5d1cb29640 4 set_mon_vals no callback set 2024-03-25T19:51:26.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.889+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f5d1401aac0 con 0x7f5d180b1730 2024-03-25T19:51:26.889 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.889+0000 7f5d1db2b640 10 monclient: get_auth_request con 0x7f5cfc06dda0 auth_method 0 2024-03-25T19:51:26.890 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.890+0000 7f5d0effd640 10 client.4685.objecter ms_handle_connect 0x7f5cfc06dda0 2024-03-25T19:51:26.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.892+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7f5d14054820 con 0x7f5d180b1730 2024-03-25T19:51:26.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.892+0000 7f5d0effd640 10 client.4685.objecter ms_dispatch 0x7f5d18000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:26.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.892+0000 7f5d0effd640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:26.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.892+0000 7f5d0effd640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:26.892 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:26.892+0000 7f5d0effd640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-25T19:51:27.179 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.179+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d14050160 con 0x7f5d180b1730 2024-03-25T19:51:27.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.179+0000 7f5d0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:27.180 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.179+0000 7f5d0effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:27.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.433+0000 7f5d20590640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.433+0000 7f5d20590640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.433+0000 7f5d20590640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:27.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.433+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d18140b30 con 0x7f5d180b1730 2024-03-25T19:51:27.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.462+0000 7f5d0effd640 1 -- 172.21.15.69:0/1898470622 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7f5d14052260 con 0x7f5d180b1730 2024-03-25T19:51:27.462 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.462+0000 7f5d0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.463 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.462+0000 7f5d0effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:27.463 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:51:27.463 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:27.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 10 librados: watch_flush enter 2024-03-25T19:51:27.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 10 librados: watch_flush exit 2024-03-25T19:51:27.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 10 client.4685.objecter successfully canceled tick 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 >> v1:172.21.15.169:6800/3072548259 conn(0x7f5cfc06dda0 legacy=0x7f5cfc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 10 monclient: shutdown 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 >> v1:172.21.15.69:6790/0 conn(0x7f5d180b1730 legacy=0x7f5d18131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 shutdown_connections 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 >> 172.21.15.69:0/1898470622 conn(0x7f5d180617a0 msgr2=0x7f5d18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 shutdown_connections 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 1 -- 172.21.15.69:0/1898470622 wait complete. 2024-03-25T19:51:27.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:27.465+0000 7f5d20590640 1 librados: shutdown 2024-03-25T19:51:27.477 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:33.900 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:11.864 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:33.968 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:11.932 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:33.995 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:11.959 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:34.012 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:11.966 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:34.019 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:11.983 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:34.020 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:11.974 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:34.027 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:11.991 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:34.050 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:12.003 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:34.077 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:12.042 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:34.283 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:34.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.398+0000 7f5cef5a3640 10 monclient: get_monmap_and_config 2024-03-25T19:51:34.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.398+0000 7f5cef5a3640 10 monclient: build_initial_monmap 2024-03-25T19:51:34.397 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.398+0000 7f5cef5a3640 10 monclient: monmap: 2024-03-25T19:51:34.397 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:34.397 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:34.397 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:51:34.398408+0000 2024-03-25T19:51:34.397 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:51:34.398408+0000 2024-03-25T19:51:34.397 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:34.398 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:34.398 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:34.398 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:51:34.398 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:51:34.398 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:34.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.398+0000 7f5cef5a3640 1 Processor -- start 2024-03-25T19:51:34.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 1 -- start start 2024-03-25T19:51:34.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 10 monclient: init 2024-03-25T19:51:34.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:34.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:34.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 10 monclient(hunting): picked mon.noname-b con 0x7f5ce80b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:34.399 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 10 monclient(hunting): picked mon.noname-a con 0x7f5ce80b1730 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:34.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 10 monclient(hunting): picked mon.noname-c con 0x7f5ce814be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:34.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80af940 con 0x7f5ce80b1730 2024-03-25T19:51:34.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80b0a70 con 0x7f5ce814be90 2024-03-25T19:51:34.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80b0d50 con 0x7f5ce80b4d60 2024-03-25T19:51:34.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:34.400 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.399+0000 7f5cef5a3640 10 monclient(hunting): authenticate will time out at 5506.667518s 2024-03-25T19:51:34.406 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.407+0000 7f5cecb3e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5ce80b1730 0x7f5ce80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60536/0 (socket says 172.21.15.69:60536) 2024-03-25T19:51:34.407 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.407+0000 7f5cecb3e640 1 -- 172.21.15.69:0/1459659153 learned_addr learned my addr 172.21.15.69:0/1459659153 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:34.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.425+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567917732 0 0) 0x7f5ce80b0a70 con 0x7f5ce814be90 2024-03-25T19:51:34.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.425+0000 7f5cdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.424 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.425+0000 7f5cdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.425+0000 7f5cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.425+0000 7f5cdffff640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:51:34.425 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.425+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd4003520 con 0x7f5ce814be90 2024-03-25T19:51:34.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.434+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802819117 0 0) 0x7f5ce80af940 con 0x7f5ce80b1730 2024-03-25T19:51:34.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.434+0000 7f5cdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.434+0000 7f5cdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.434+0000 7f5cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.434+0000 7f5cdffff640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:51:34.433 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.434+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce80b0a70 con 0x7f5ce80b1730 2024-03-25T19:51:34.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.444+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286651300 0 0) 0x7f5ce80b0d50 con 0x7f5ce80b4d60 2024-03-25T19:51:34.443 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.444+0000 7f5cdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.444+0000 7f5cdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.444+0000 7f5cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.444+0000 7f5cdffff640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:51:34.444 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.444+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce80af940 con 0x7f5ce80b4d60 2024-03-25T19:51:34.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756218652 0 0) 0x7f5ce80b0a70 con 0x7f5ce80b1730 2024-03-25T19:51:34.447 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 >> v1:172.21.15.69:6790/0 conn(0x7f5ce80b4d60 legacy=0x7f5ce813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 >> v1:172.21.15.169:6789/0 conn(0x7f5ce814be90 legacy=0x7f5ce814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 1 monclient: found mon.noname-a 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce80b1030 con 0x7f5ce80b1730 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 10 monclient: _finish_auth 0 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5cd8001670 con 0x7f5ce80b1730 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cdffff640 10 monclient: dump: 2024-03-25T19:51:34.448 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cef5a3640 5 monclient: authenticate success, global_id 4719 2024-03-25T19:51:34.449 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.448+0000 7f5cef5a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:34.452 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.453+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cd80017d0 con 0x7f5ce80b1730 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.453+0000 7f5cdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.453+0000 7f5cdffff640 1 -- 172.21.15.69:0/1459659153 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5cd8001be0 con 0x7f5ce80b1730 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.453+0000 7f5cef5a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.453+0000 7f5cef5a3640 4 set_mon_vals no callback set 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.453+0000 7f5cdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.454+0000 7f5cdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.454+0000 7f5cdffff640 10 monclient: dump: 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:34.453 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:34.454 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:34.454 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:34.454 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:34.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.454+0000 7f5cef5a3640 10 monclient: shutdown 2024-03-25T19:51:34.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.454+0000 7f5cef5a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:34.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.454+0000 7f5cef5a3640 1 -- 172.21.15.69:0/1459659153 >> v1:172.21.15.69:6789/0 conn(0x7f5ce80b1730 legacy=0x7f5ce80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.454+0000 7f5cef5a3640 1 -- 172.21.15.69:0/1459659153 shutdown_connections 2024-03-25T19:51:34.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.454+0000 7f5cef5a3640 1 -- 172.21.15.69:0/1459659153 >> 172.21.15.69:0/1459659153 conn(0x7f5ce80617a0 msgr2=0x7f5ce8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:34.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.455+0000 7f5cef5a3640 1 -- 172.21.15.69:0/1459659153 shutdown_connections 2024-03-25T19:51:34.455 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.455+0000 7f5cef5a3640 1 -- 172.21.15.69:0/1459659153 wait complete. 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.455+0000 7f5cef5a3640 10 monclient: build_initial_monmap 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.455+0000 7f5cef5a3640 10 monclient: monmap: 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.456+0000 7f5cef5a3640 1 librados: starting msgr at 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.456+0000 7f5cef5a3640 1 librados: starting objecter 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.456+0000 7f5cef5a3640 1 Processor -- start 2024-03-25T19:51:34.456 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.456+0000 7f5cef5a3640 1 -- start start 2024-03-25T19:51:34.457 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.456+0000 7f5cef5a3640 1 librados: setting wanted keys 2024-03-25T19:51:34.457 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.456+0000 7f5cef5a3640 1 librados: calling monclient init 2024-03-25T19:51:34.457 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.456+0000 7f5cef5a3640 10 monclient: init 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 10 monclient(hunting): picked mon.noname-c con 0x7f5ce80b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 10 monclient(hunting): picked mon.noname-b con 0x7f5ce80b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 10 monclient(hunting): picked mon.noname-a con 0x7f5ce814be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce81d7b10 con 0x7f5ce814be90 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce8136830 con 0x7f5ce80b4d60 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce81d7730 con 0x7f5ce80b1730 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:34.458 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.457+0000 7f5cef5a3640 10 monclient(hunting): authenticate will time out at 5506.724887s 2024-03-25T19:51:34.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.466+0000 7f5cecb3e640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5ce80b4d60 0x7f5ce81d8030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:59376/0 (socket says 172.21.15.69:59376) 2024-03-25T19:51:34.465 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.466+0000 7f5cecb3e640 1 -- 172.21.15.69:0/2332776392 learned_addr learned my addr 172.21.15.69:0/2332776392 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:34.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.467+0000 7f5cddffb640 10 client.?.objecter ms_handle_connect 0x7f5ce80b4d60 2024-03-25T19:51:34.466 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.467+0000 7f5cddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.472+0000 7f5cddffb640 10 client.?.objecter ms_handle_connect 0x7f5ce814be90 2024-03-25T19:51:34.471 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.472+0000 7f5cddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.486+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109451972 0 0) 0x7f5ce81d7b10 con 0x7f5ce814be90 2024-03-25T19:51:34.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.486+0000 7f5cddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.486+0000 7f5cddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.486+0000 7f5cddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.486+0000 7f5cddffb640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:51:34.485 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.486+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc004eb0 con 0x7f5ce814be90 2024-03-25T19:51:34.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.488+0000 7f5cddffb640 10 client.?.objecter ms_handle_connect 0x7f5ce80b1730 2024-03-25T19:51:34.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.488+0000 7f5cddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.493 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.494+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402788927 0 0) 0x7f5ce8136830 con 0x7f5ce80b4d60 2024-03-25T19:51:34.493 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.494+0000 7f5cddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.493 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.494+0000 7f5cddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.493 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.494+0000 7f5cddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.494 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.494+0000 7f5cddffb640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:51:34.494 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.494+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce81d7b10 con 0x7f5ce80b4d60 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.505+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4182255936 0 0) 0x7f5ce81d7b10 con 0x7f5ce80b4d60 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.505+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ce8136830 con 0x7f5ce80b4d60 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.505+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5cd8001ba0 con 0x7f5ce80b4d60 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.505+0000 7f5cddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.505+0000 7f5cddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.505+0000 7f5cddffb640 10 monclient(hunting): dump: 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.504 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.505 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:34.505 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:34.505 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:34.505 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:34.505 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:34.505 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.505+0000 7f5cddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:34.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.512+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797137691 0 0) 0x7f5ce81d7730 con 0x7f5ce80b1730 2024-03-25T19:51:34.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.512+0000 7f5cddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.512+0000 7f5cddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.511 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.512+0000 7f5cddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.512 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.512+0000 7f5cddffb640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:51:34.512 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.512+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce81d7b10 con 0x7f5ce80b1730 2024-03-25T19:51:34.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.514+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338001122 0 0) 0x7f5ccc004eb0 con 0x7f5ce814be90 2024-03-25T19:51:34.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.514+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ce81d7730 con 0x7f5ce814be90 2024-03-25T19:51:34.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.514+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5ce0001af0 con 0x7f5ce814be90 2024-03-25T19:51:34.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.514+0000 7f5cddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.514+0000 7f5cddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:34.513 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.514+0000 7f5cddffb640 10 monclient(hunting): dump: 2024-03-25T19:51:34.513 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:34.514 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:34.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3374511719 0 0) 0x7f5ce8136830 con 0x7f5ce80b4d60 2024-03-25T19:51:34.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 >> v1:172.21.15.69:6790/0 conn(0x7f5ce80b1730 legacy=0x7f5ce8143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 >> v1:172.21.15.69:6789/0 conn(0x7f5ce814be90 legacy=0x7f5ce8137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.522 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 1 monclient: found mon.b 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce81d78c0 con 0x7f5ce80b4d60 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 10 monclient: _renew_subs 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce8137a10 con 0x7f5ce80b4d60 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 10 monclient: _renew_subs 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce813bb00 con 0x7f5ce80b4d60 2024-03-25T19:51:34.523 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.523+0000 7f5cef5a3640 1 librados: init done 2024-03-25T19:51:34.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.524+0000 7f5cef5a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.524+0000 7f5cef5a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.524+0000 7f5cef5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:34.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.524+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce81d7ca0 con 0x7f5ce80b4d60 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.820+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cd8004a90 con 0x7f5ce80b4d60 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.820+0000 7f5cddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.820+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f5cd8004da0 con 0x7f5ce80b4d60 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.820+0000 7f5cddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.820+0000 7f5cddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.820+0000 7f5cddffb640 10 monclient: dump: 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:34.820 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.821 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:34.821 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.821 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:34.821 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:34.821 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:34.821 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:34.821 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:34.821 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.820+0000 7f5cdf7fe640 4 set_mon_vals no callback set 2024-03-25T19:51:34.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.821+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f5cd801ad70 con 0x7f5ce80b4d60 2024-03-25T19:51:34.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.822+0000 7f5ced33f640 10 monclient: get_auth_request con 0x7f5ccc06df10 auth_method 0 2024-03-25T19:51:34.822 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.822+0000 7f5cddffb640 10 client.4693.objecter ms_handle_connect 0x7f5ccc06df10 2024-03-25T19:51:34.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.829+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7f5cd8054950 con 0x7f5ce80b4d60 2024-03-25T19:51:34.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.829+0000 7f5cddffb640 10 client.4693.objecter ms_dispatch 0x7f5ce8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:34.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.829+0000 7f5cddffb640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:34.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.829+0000 7f5cddffb640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:34.828 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.829+0000 7f5cddffb640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T19:51:34.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.830+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5cd8050460 con 0x7f5ce80b4d60 2024-03-25T19:51:34.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.830+0000 7f5cddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.829 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:34.830+0000 7f5cddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:35.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.082+0000 7f5cef5a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.082+0000 7f5cef5a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.082+0000 7f5cef5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:35.081 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.082+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ce80627b0 con 0x7f5ce80b4d60 2024-03-25T19:51:35.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.206+0000 7f5cddffb640 1 -- 172.21.15.69:0/2332776392 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7f5cd8052390 con 0x7f5ce80b4d60 2024-03-25T19:51:35.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.206+0000 7f5cddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.206 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.206+0000 7f5cddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:35.206 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:51:35.206 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:35.208 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 10 librados: watch_flush enter 2024-03-25T19:51:35.208 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 10 librados: watch_flush exit 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 10 client.4693.objecter successfully canceled tick 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 >> v1:172.21.15.169:6800/3072548259 conn(0x7f5ccc06df10 legacy=0x7f5ccc0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 10 monclient: shutdown 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 >> v1:172.21.15.169:6789/0 conn(0x7f5ce80b4d60 legacy=0x7f5ce81d8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 shutdown_connections 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 >> 172.21.15.69:0/2332776392 conn(0x7f5ce80617a0 msgr2=0x7f5ce813c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 shutdown_connections 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 1 -- 172.21.15.69:0/2332776392 wait complete. 2024-03-25T19:51:35.209 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:35.209+0000 7f5cef5a3640 1 librados: shutdown 2024-03-25T19:51:35.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:42.028 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.139+0000 7ff4a2ce4640 10 monclient: get_monmap_and_config 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.139+0000 7ff4a2ce4640 10 monclient: build_initial_monmap 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.139+0000 7ff4a2ce4640 10 monclient: monmap: 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:51:42.140382+0000 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:51:42.140382+0000 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:42.139 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:42.140 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:42.140 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:51:42.140 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:51:42.140 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 1 Processor -- start 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 1 -- start start 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 10 monclient: init 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 10 monclient(hunting): picked mon.noname-a con 0x7ff49c0a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 10 monclient(hunting): picked mon.noname-c con 0x7ff49c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 10 monclient(hunting): picked mon.noname-b con 0x7ff49c1430c0 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c142520 con 0x7ff49c0a98f0 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c0b28a0 con 0x7ff49c13f710 2024-03-25T19:51:42.141 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c0b2a30 con 0x7ff49c1430c0 2024-03-25T19:51:42.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:42.142 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.140+0000 7ff4a2ce4640 10 monclient(hunting): authenticate will time out at 5514.409591s 2024-03-25T19:51:42.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.150+0000 7ff4a1281640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7ff49c1430c0 0x7ff49c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:39034/0 (socket says 172.21.15.69:39034) 2024-03-25T19:51:42.150 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.150+0000 7ff4a1281640 1 -- 172.21.15.69:0/2429334888 learned_addr learned my addr 172.21.15.69:0/2429334888 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:42.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.163+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717132815 0 0) 0x7ff49c0b28a0 con 0x7ff49c13f710 2024-03-25T19:51:42.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.163+0000 7ff49b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.163+0000 7ff49b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.163 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.163+0000 7ff49b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.164 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.163+0000 7ff49b7fe640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:51:42.164 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.163+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff48c003520 con 0x7ff49c13f710 2024-03-25T19:51:42.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.168+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170880616 0 0) 0x7ff49c0b2a30 con 0x7ff49c1430c0 2024-03-25T19:51:42.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.168+0000 7ff49b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.168+0000 7ff49b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.168+0000 7ff49b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.168+0000 7ff49b7fe640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:51:42.168 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.168+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff49c0b28a0 con 0x7ff49c1430c0 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.176+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2824123665 0 0) 0x7ff48c003520 con 0x7ff49c13f710 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.176+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 >> v1:172.21.15.69:6790/0 conn(0x7ff49c1430c0 legacy=0x7ff49c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.176+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 >> v1:172.21.15.69:6789/0 conn(0x7ff49c0a98f0 legacy=0x7ff49c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.176+0000 7ff49b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.176+0000 7ff49b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c0b44c0 con 0x7ff49c13f710 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:42.177 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff488001690 con 0x7ff49c13f710 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff49b7fe640 10 monclient: dump: 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff4a2ce4640 5 monclient: authenticate success, global_id 4696 2024-03-25T19:51:42.178 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.177+0000 7ff4a2ce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:42.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.182+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4880017f0 con 0x7ff49c13f710 2024-03-25T19:51:42.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.182+0000 7ff49b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:42.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.182+0000 7ff49b7fe640 1 -- 172.21.15.69:0/2429334888 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff488001c00 con 0x7ff49c13f710 2024-03-25T19:51:42.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.182+0000 7ff4a2ce4640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:42.182 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.182+0000 7ff4a2ce4640 4 set_mon_vals no callback set 2024-03-25T19:51:42.183 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.182+0000 7ff49b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.183 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.182+0000 7ff49b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:42.183 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.182+0000 7ff49b7fe640 10 monclient: dump: 2024-03-25T19:51:42.183 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:42.183 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:42.183 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:42.183 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.183+0000 7ff4a2ce4640 10 monclient: shutdown 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.183+0000 7ff4a2ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.183+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/2429334888 >> v1:172.21.15.169:6789/0 conn(0x7ff49c13f710 legacy=0x7ff49c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.183+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/2429334888 shutdown_connections 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.183+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/2429334888 >> 172.21.15.69:0/2429334888 conn(0x7ff49c0b7ea0 msgr2=0x7ff49c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.183+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/2429334888 shutdown_connections 2024-03-25T19:51:42.184 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.183+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/2429334888 wait complete. 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 10 monclient: build_initial_monmap 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 10 monclient: monmap: 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:51:42.186 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 1 librados: starting msgr at 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 1 librados: starting objecter 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 1 Processor -- start 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 1 -- start start 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 1 librados: setting wanted keys 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 1 librados: calling monclient init 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.184+0000 7ff4a2ce4640 10 monclient: init 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 10 monclient(hunting): picked mon.noname-c con 0x7ff49c0a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 10 monclient(hunting): picked mon.noname-a con 0x7ff49c13f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 10 monclient(hunting): picked mon.noname-b con 0x7ff49c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:42.187 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c136a30 con 0x7ff49c13f710 2024-03-25T19:51:42.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c136bc0 con 0x7ff49c1430c0 2024-03-25T19:51:42.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c136d50 con 0x7ff49c0a98f0 2024-03-25T19:51:42.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.185+0000 7ff4a2ce4640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:42.188 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.186+0000 7ff4a2ce4640 10 monclient(hunting): authenticate will time out at 5514.454612s 2024-03-25T19:51:42.198 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.198+0000 7ff4a1281640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ff49c1430c0 0x7ff49c1e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:43474/0 (socket says 172.21.15.69:43474) 2024-03-25T19:51:42.198 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.198+0000 7ff4a1281640 1 -- 172.21.15.69:0/1073868514 learned_addr learned my addr 172.21.15.69:0/1073868514 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:42.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.205+0000 7ff4997fa640 10 client.?.objecter ms_handle_connect 0x7ff49c1430c0 2024-03-25T19:51:42.205 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.205+0000 7ff4997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.211+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497627482 0 0) 0x7ff49c136bc0 con 0x7ff49c1430c0 2024-03-25T19:51:42.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.211+0000 7ff4997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.211+0000 7ff4997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.211+0000 7ff4997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.211+0000 7ff4997fa640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:51:42.211 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.211+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff480004eb0 con 0x7ff49c1430c0 2024-03-25T19:51:42.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.215+0000 7ff4997fa640 10 client.?.objecter ms_handle_connect 0x7ff49c13f710 2024-03-25T19:51:42.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.215+0000 7ff4997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.220+0000 7ff4997fa640 10 client.?.objecter ms_handle_connect 0x7ff49c0a98f0 2024-03-25T19:51:42.220 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.220+0000 7ff4997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.227+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3848871556 0 0) 0x7ff480004eb0 con 0x7ff49c1430c0 2024-03-25T19:51:42.227 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.227+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff49c136bc0 con 0x7ff49c1430c0 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.227+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff494001c10 con 0x7ff49c1430c0 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.227+0000 7ff4997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.227+0000 7ff4997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.227+0000 7ff4997fa640 10 monclient(hunting): dump: 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:42.228 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.227+0000 7ff4997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:42.233 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.233+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918415768 0 0) 0x7ff49c136d50 con 0x7ff49c0a98f0 2024-03-25T19:51:42.233 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.233+0000 7ff4997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.233 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.233+0000 7ff4997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.233+0000 7ff4997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.233+0000 7ff4997fa640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:51:42.234 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.233+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff480004eb0 con 0x7ff49c0a98f0 2024-03-25T19:51:42.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.235+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090431607 0 0) 0x7ff49c136a30 con 0x7ff49c13f710 2024-03-25T19:51:42.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.235+0000 7ff4997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.235+0000 7ff4997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.235+0000 7ff4997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.235+0000 7ff4997fa640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:51:42.235 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.235+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff49c136d50 con 0x7ff49c13f710 2024-03-25T19:51:42.236 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 591748941 0 0) 0x7ff49c136bc0 con 0x7ff49c1430c0 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 >> v1:172.21.15.69:6790/0 conn(0x7ff49c0a98f0 legacy=0x7ff49c1dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 >> v1:172.21.15.69:6789/0 conn(0x7ff49c13f710 legacy=0x7ff49c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 1 monclient: found mon.b 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c0b7110 con 0x7ff49c1430c0 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 5 monclient: authenticate success, global_id 4699 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 10 monclient: _renew_subs 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c0b7380 con 0x7ff49c1430c0 2024-03-25T19:51:42.237 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 10 client.4699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:42.238 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 10 monclient: _renew_subs 2024-03-25T19:51:42.238 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:42.238 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c139600 con 0x7ff49c1430c0 2024-03-25T19:51:42.238 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.236+0000 7ff4a2ce4640 1 librados: init done 2024-03-25T19:51:42.238 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.237+0000 7ff4a2ce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.238 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.237+0000 7ff4a2ce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.238 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.237+0000 7ff4a2ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:42.239 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.237+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c0b3ef0 con 0x7ff49c1430c0 2024-03-25T19:51:42.240 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.240+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff494004b60 con 0x7ff49c1430c0 2024-03-25T19:51:42.240 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.240+0000 7ff4997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:42.240 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.240+0000 7ff49affd640 4 set_mon_vals no callback set 2024-03-25T19:51:42.519 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.519+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7ff494004e70 con 0x7ff49c1430c0 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.519+0000 7ff4997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.519+0000 7ff4997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.519+0000 7ff4997fa640 10 monclient: dump: 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:42.520 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:42.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.520+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7ff49401ae40 con 0x7ff49c1430c0 2024-03-25T19:51:42.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.520+0000 7ff4a0a80640 10 monclient: get_auth_request con 0x7ff48006de80 auth_method 0 2024-03-25T19:51:42.521 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.521+0000 7ff4997fa640 10 client.4699.objecter ms_handle_connect 0x7ff48006de80 2024-03-25T19:51:42.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.524+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7ff494054bc0 con 0x7ff49c1430c0 2024-03-25T19:51:42.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.524+0000 7ff4997fa640 10 client.4699.objecter ms_dispatch 0x7ff49c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:42.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.524+0000 7ff4997fa640 3 client.4699.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:42.524 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.524+0000 7ff4997fa640 3 client.4699.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:42.525 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.524+0000 7ff4997fa640 20 client.4699.objecter dump_active .. 0 homeless 2024-03-25T19:51:42.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.528+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff494050500 con 0x7ff49c1430c0 2024-03-25T19:51:42.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.528+0000 7ff4997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.528 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.528+0000 7ff4997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:42.768 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.768+0000 7ff4a2ce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.768+0000 7ff4a2ce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.768+0000 7ff4a2ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:42.769 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.768+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff49c137680 con 0x7ff49c1430c0 2024-03-25T19:51:42.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.953+0000 7ff4997fa640 1 -- 172.21.15.69:0/1073868514 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7ff49c137680 con 0x7ff49c1430c0 2024-03-25T19:51:42.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.953+0000 7ff4997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.953+0000 7ff4997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:42.954 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:51:42.954 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:42.956 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 10 librados: watch_flush enter 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 10 librados: watch_flush exit 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 20 client.4699.objecter shutdown clearing up homeless session... 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 10 client.4699.objecter successfully canceled tick 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 >> v1:172.21.15.169:6800/3072548259 conn(0x7ff48006de80 legacy=0x7ff480070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 10 monclient: shutdown 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 >> v1:172.21.15.169:6789/0 conn(0x7ff49c1430c0 legacy=0x7ff49c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 shutdown_connections 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 >> 172.21.15.69:0/1073868514 conn(0x7ff49c0b7ea0 msgr2=0x7ff49c13c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 shutdown_connections 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 1 -- 172.21.15.69:0/1073868514 wait complete. 2024-03-25T19:51:42.957 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:42.956+0000 7ff4a2ce4640 1 librados: shutdown 2024-03-25T19:51:42.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:45.156 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:23.120 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:45.160 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:23.124 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:45.179 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:23.134 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:45.182 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:23.146 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:45.182 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:23.136 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:45.184 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:23.148 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:45.185 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:23.149 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:45.198 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:23.152 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:45.200 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:23.164 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:49.773 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:49.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.884+0000 7f92722c9640 10 monclient: get_monmap_and_config 2024-03-25T19:51:49.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.884+0000 7f92722c9640 10 monclient: build_initial_monmap 2024-03-25T19:51:49.884 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.884+0000 7f92722c9640 10 monclient: monmap: 2024-03-25T19:51:49.884 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:49.884 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:49.884 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:51:49.885532+0000 2024-03-25T19:51:49.884 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:51:49.885532+0000 2024-03-25T19:51:49.885 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:49.885 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:49.885 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:49.885 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:51:49.885 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:51:49.885 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:49.886 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.885+0000 7f92722c9640 1 Processor -- start 2024-03-25T19:51:49.886 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.885+0000 7f92722c9640 1 -- start start 2024-03-25T19:51:49.886 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.885+0000 7f92722c9640 10 monclient: init 2024-03-25T19:51:49.886 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:49.886 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:49.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 10 monclient(hunting): picked mon.noname-b con 0x7f926c13fc60 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:49.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 10 monclient(hunting): picked mon.noname-a con 0x7f926c070570 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:49.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 10 monclient(hunting): picked mon.noname-c con 0x7f926c0727a0 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:49.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c13c9f0 con 0x7f926c070570 2024-03-25T19:51:49.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c06fac0 con 0x7f926c0727a0 2024-03-25T19:51:49.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c0715d0 con 0x7f926c13fc60 2024-03-25T19:51:49.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:49.887 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.886+0000 7f92722c9640 10 monclient(hunting): authenticate will time out at 5522.154892s 2024-03-25T19:51:49.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.895+0000 7f9270866640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f926c0727a0 0x7f926c14a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:48132/0 (socket says 172.21.15.69:48132) 2024-03-25T19:51:49.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.895+0000 7f9270866640 1 -- 172.21.15.69:0/858579967 learned_addr learned my addr 172.21.15.69:0/858579967 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:49.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.912+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697248382 0 0) 0x7f926c06fac0 con 0x7f926c0727a0 2024-03-25T19:51:49.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.912+0000 7f926affd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.912+0000 7f926affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.912+0000 7f926affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.912+0000 7f926affd640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:51:49.913 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.912+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9254005730 con 0x7f926c0727a0 2024-03-25T19:51:49.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.925+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936378268 0 0) 0x7f926c0715d0 con 0x7f926c13fc60 2024-03-25T19:51:49.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.925+0000 7f926affd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.925+0000 7f926affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.925+0000 7f926affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.925 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.925+0000 7f926affd640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:51:49.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.925+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c06fac0 con 0x7f926c13fc60 2024-03-25T19:51:49.926 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.925+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4009571385 0 0) 0x7f9254005730 con 0x7f926c0727a0 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.925+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 >> v1:172.21.15.69:6790/0 conn(0x7f926c13fc60 legacy=0x7f926c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 >> v1:172.21.15.69:6789/0 conn(0x7f926c070570 legacy=0x7f926c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 1 monclient: found mon.noname-c 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0718b0 con 0x7f926c0727a0 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 10 monclient: _finish_auth 0 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9264001a10 con 0x7f926c0727a0 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f926affd640 10 monclient: dump: 2024-03-25T19:51:49.927 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f92722c9640 5 monclient: authenticate success, global_id 4702 2024-03-25T19:51:49.928 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.926+0000 7f92722c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.949+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9264001bf0 con 0x7f926c0727a0 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.949+0000 7f926affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.949+0000 7f926affd640 1 -- 172.21.15.69:0/858579967 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9264002000 con 0x7f926c0727a0 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.949+0000 7f926affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.949+0000 7f926affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.949+0000 7f926affd640 10 monclient: dump: 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:49.949 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:49.950 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:49.950 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:49.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.949+0000 7f92722c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:49.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.949+0000 7f92722c9640 4 set_mon_vals no callback set 2024-03-25T19:51:49.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.950+0000 7f92722c9640 10 monclient: shutdown 2024-03-25T19:51:49.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.950+0000 7f92722c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:49.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.950+0000 7f92722c9640 1 -- 172.21.15.69:0/858579967 >> v1:172.21.15.169:6789/0 conn(0x7f926c0727a0 legacy=0x7f926c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.950+0000 7f92722c9640 1 -- 172.21.15.69:0/858579967 shutdown_connections 2024-03-25T19:51:49.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.950+0000 7f92722c9640 1 -- 172.21.15.69:0/858579967 >> 172.21.15.69:0/858579967 conn(0x7f926c0b4a80 msgr2=0x7f926c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:49.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.950+0000 7f92722c9640 1 -- 172.21.15.69:0/858579967 shutdown_connections 2024-03-25T19:51:49.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.950+0000 7f92722c9640 1 -- 172.21.15.69:0/858579967 wait complete. 2024-03-25T19:51:49.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.951+0000 7f92722c9640 10 monclient: build_initial_monmap 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.951+0000 7f92722c9640 10 monclient: monmap: 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.951+0000 7f92722c9640 1 librados: starting msgr at 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.951+0000 7f92722c9640 1 librados: starting objecter 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.951+0000 7f92722c9640 1 Processor -- start 2024-03-25T19:51:49.952 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.952+0000 7f92722c9640 1 -- start start 2024-03-25T19:51:49.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.952+0000 7f92722c9640 1 librados: setting wanted keys 2024-03-25T19:51:49.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.952+0000 7f92722c9640 1 librados: calling monclient init 2024-03-25T19:51:49.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.952+0000 7f92722c9640 10 monclient: init 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 10 monclient(hunting): picked mon.noname-a con 0x7f926c070570 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 10 monclient(hunting): picked mon.noname-c con 0x7f926c0727a0 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 10 monclient(hunting): picked mon.noname-b con 0x7f926c13fc60 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c1c86f0 con 0x7f926c070570 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c141ec0 con 0x7f926c13fc60 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c143360 con 0x7f926c0727a0 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:49.954 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.953+0000 7f92722c9640 10 monclient(hunting): authenticate will time out at 5522.222056s 2024-03-25T19:51:49.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.960+0000 7f926b7fe640 1 --1- >> v1:172.21.15.69:6790/0 conn(0x7f926c0727a0 0x7f926c1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6790/0 says I am v1:172.21.15.69:43908/0 (socket says 172.21.15.69:43908) 2024-03-25T19:51:49.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.960+0000 7f926b7fe640 1 -- 172.21.15.69:0/1111794598 learned_addr learned my addr 172.21.15.69:0/1111794598 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:49.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.961+0000 7f9268ff9640 10 client.?.objecter ms_handle_connect 0x7f926c0727a0 2024-03-25T19:51:49.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.961+0000 7f9268ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:49.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.966+0000 7f9268ff9640 10 client.?.objecter ms_handle_connect 0x7f926c070570 2024-03-25T19:51:49.966 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.966+0000 7f9268ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:49.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.967+0000 7f9268ff9640 10 client.?.objecter ms_handle_connect 0x7f926c13fc60 2024-03-25T19:51:49.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.967+0000 7f9268ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:49.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.967+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775484346 0 0) 0x7f926c143360 con 0x7f926c0727a0 2024-03-25T19:51:49.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.967+0000 7f9268ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.967+0000 7f9268ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.967+0000 7f9268ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.967+0000 7f9268ff9640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:51:49.967 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.967+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9258004eb0 con 0x7f926c0727a0 2024-03-25T19:51:49.971 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.971+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540752016 0 0) 0x7f926c1c86f0 con 0x7f926c070570 2024-03-25T19:51:49.971 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.971+0000 7f9268ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.971 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.971+0000 7f9268ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.971 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.971+0000 7f9268ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.971 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.971+0000 7f9268ff9640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:51:49.971 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.971+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c143360 con 0x7f926c070570 2024-03-25T19:51:49.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.976+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536934558 0 0) 0x7f926c141ec0 con 0x7f926c13fc60 2024-03-25T19:51:49.976 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.976+0000 7f9268ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.976+0000 7f9268ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.976+0000 7f9268ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.976+0000 7f9268ff9640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:51:49.977 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.976+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c1c86f0 con 0x7f926c13fc60 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.978+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310778969 0 0) 0x7f9258004eb0 con 0x7f926c0727a0 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.978+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f926c141ec0 con 0x7f926c0727a0 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.978+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f92600019d0 con 0x7f926c0727a0 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.978+0000 7f9268ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.978+0000 7f9268ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.978+0000 7f9268ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:49.978 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:49.979 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:49.979 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.979 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:49.979 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:49.979 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:49.979 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:49.979 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:49.979 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.978+0000 7f9268ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:49.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.981+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 533178325 0 0) 0x7f926c143360 con 0x7f926c070570 2024-03-25T19:51:49.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.981+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9258004eb0 con 0x7f926c070570 2024-03-25T19:51:49.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.981+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f925c001810 con 0x7f926c070570 2024-03-25T19:51:49.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.981+0000 7f9268ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.981 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.981+0000 7f9268ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.981+0000 7f9268ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:49.982 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:49.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 749525605 0 0) 0x7f926c141ec0 con 0x7f926c0727a0 2024-03-25T19:51:49.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 >> v1:172.21.15.169:6789/0 conn(0x7f926c13fc60 legacy=0x7f926c135bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 >> v1:172.21.15.69:6789/0 conn(0x7f926c070570 legacy=0x7f926c1412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:49.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 1 monclient: found mon.c 2024-03-25T19:51:49.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:49.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:49.983 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c139540 con 0x7f926c0727a0 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f9268ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 5 monclient: authenticate success, global_id 4703 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 10 monclient: _renew_subs 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c1397f0 con 0x7f926c0727a0 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 10 monclient: _renew_subs 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c1364c0 con 0x7f926c0727a0 2024-03-25T19:51:49.984 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.983+0000 7f92722c9640 1 librados: init done 2024-03-25T19:51:49.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.984+0000 7f92722c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:49.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.984+0000 7f92722c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:49.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.984+0000 7f92722c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:49.985 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:49.984+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c0627b0 con 0x7f926c0727a0 2024-03-25T19:51:50.280 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.280+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9260001b30 con 0x7f926c0727a0 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.280+0000 7f9268ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.280+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f9260003a60 con 0x7f926c0727a0 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.280+0000 7f9268ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.280+0000 7f9268ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.280+0000 7f9268ff9640 10 monclient: dump: 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:50.281 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:50.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.280+0000 7f926a7fc640 4 set_mon_vals no callback set 2024-03-25T19:51:50.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.281+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f9260003c80 con 0x7f926c0727a0 2024-03-25T19:51:50.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.282+0000 7f926bfff640 10 monclient: get_auth_request con 0x7f925806dfb0 auth_method 0 2024-03-25T19:51:50.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.282+0000 7f9268ff9640 10 client.4703.objecter ms_handle_connect 0x7f925806dfb0 2024-03-25T19:51:50.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.288+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7f9260054800 con 0x7f926c0727a0 2024-03-25T19:51:50.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.289+0000 7f9268ff9640 10 client.4703.objecter ms_dispatch 0x7f926c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:50.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.289+0000 7f9268ff9640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:50.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.289+0000 7f9268ff9640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:50.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.289+0000 7f9268ff9640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T19:51:50.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.289+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9260050310 con 0x7f926c0727a0 2024-03-25T19:51:50.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.289+0000 7f9268ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:50.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.289+0000 7f9268ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:50.535 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.535+0000 7f92722c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.536 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.535+0000 7f92722c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.536 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.535+0000 7f92722c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.69:6790/0 2024-03-25T19:51:50.536 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.535+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 --> v1:172.21.15.69:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f926c0af160 con 0x7f926c0727a0 2024-03-25T19:51:50.553 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.553+0000 7f9268ff9640 1 -- 172.21.15.69:0/1111794598 <== mon.2 v1:172.21.15.69:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7f9260052240 con 0x7f926c0727a0 2024-03-25T19:51:50.553 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.553+0000 7f9268ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.553 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.553+0000 7f9268ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:50.554 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:51:50.554 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 10 librados: watch_flush enter 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 10 librados: watch_flush exit 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 10 client.4703.objecter successfully canceled tick 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 >> v1:172.21.15.169:6800/3072548259 conn(0x7f925806dfb0 legacy=0x7f9258070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 10 monclient: shutdown 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 >> v1:172.21.15.69:6790/0 conn(0x7f926c0727a0 legacy=0x7f926c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 shutdown_connections 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.556+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 >> 172.21.15.69:0/1111794598 conn(0x7f926c0b4a80 msgr2=0x7f926c0afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:50.557 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.557+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 shutdown_connections 2024-03-25T19:51:50.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.557+0000 7f92722c9640 1 -- 172.21.15.69:0/1111794598 wait complete. 2024-03-25T19:51:50.558 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:50.557+0000 7f92722c9640 1 librados: shutdown 2024-03-25T19:51:50.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:53.993 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:31.957 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:54.003 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:31.967 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:54.038 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:32.002 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:54.054 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:32.018 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:54.056 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:32.010 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:54.060 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:32.024 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:54.063 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:32.017 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:54.097 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:32.060 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:54.097 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:32.051 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.372 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:57.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.487+0000 7fb31924a640 10 monclient: get_monmap_and_config 2024-03-25T19:51:57.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.487+0000 7fb31924a640 10 monclient: build_initial_monmap 2024-03-25T19:51:57.487 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.487+0000 7fb31924a640 10 monclient: monmap: 2024-03-25T19:51:57.487 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:57.487 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:57.487 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:51:57.488680+0000 2024-03-25T19:51:57.487 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:51:57.488680+0000 2024-03-25T19:51:57.488 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:57.488 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:57.488 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:57.488 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:51:57.488 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:51:57.488 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:57.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 1 Processor -- start 2024-03-25T19:51:57.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 1 -- start start 2024-03-25T19:51:57.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 10 monclient: init 2024-03-25T19:51:57.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:57.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:57.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 10 monclient(hunting): picked mon.noname-a con 0x7fb3140b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:57.489 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 10 monclient(hunting): picked mon.noname-c con 0x7fb3140b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:57.490 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 10 monclient(hunting): picked mon.noname-b con 0x7fb31414be90 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:57.490 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.488+0000 7fb31924a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140af940 con 0x7fb3140b4d60 2024-03-25T19:51:57.490 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.489+0000 7fb31924a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140b0a70 con 0x7fb3140b1730 2024-03-25T19:51:57.490 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.489+0000 7fb31924a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140b0d50 con 0x7fb31414be90 2024-03-25T19:51:57.490 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.489+0000 7fb31924a640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:57.490 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.489+0000 7fb31924a640 10 monclient(hunting): authenticate will time out at 5529.757649s 2024-03-25T19:51:57.494 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.494+0000 7fb31259c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb3140b1730 0x7fb3140b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:45312/0 (socket says 172.21.15.69:45312) 2024-03-25T19:51:57.494 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.494+0000 7fb31259c640 1 -- 172.21.15.69:0/748419173 learned_addr learned my addr 172.21.15.69:0/748419173 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:57.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.529+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069292075 0 0) 0x7fb3140af940 con 0x7fb3140b4d60 2024-03-25T19:51:57.529 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.530+0000 7fb311d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.530+0000 7fb311d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.530+0000 7fb311d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.530+0000 7fb311d9b640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:51:57.530 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.530+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc003520 con 0x7fb3140b4d60 2024-03-25T19:51:57.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191151408 0 0) 0x7fb2fc003520 con 0x7fb3140b4d60 2024-03-25T19:51:57.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 >> v1:172.21.15.69:6790/0 conn(0x7fb31414be90 legacy=0x7fb31414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 >> v1:172.21.15.169:6789/0 conn(0x7fb3140b1730 legacy=0x7fb3140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:51:57.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 1 monclient: found mon.noname-a 2024-03-25T19:51:57.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:57.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.69:6789/0 2024-03-25T19:51:57.597 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140b1030 con 0x7fb3140b4d60 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 10 monclient: _finish_auth 0 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb304001690 con 0x7fb3140b4d60 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb311d9b640 10 monclient: dump: 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:57.598 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:57.599 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:57.599 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:57.599 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:57.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb31924a640 5 monclient: authenticate success, global_id 4734 2024-03-25T19:51:57.599 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.596+0000 7fb31924a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 <== mon.0 v1:172.21.15.69:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3040017f0 con 0x7fb3140b4d60 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb311d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb311d9b640 1 -- 172.21.15.69:0/748419173 <== mon.0 v1:172.21.15.69:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb304001c00 con 0x7fb3140b4d60 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb31924a640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb31924a640 4 set_mon_vals no callback set 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb311d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb311d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb311d9b640 10 monclient: dump: 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.616 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.617 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.617 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:57.617 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:57.617 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:57.617 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:57.617 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:57.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb31924a640 10 monclient: shutdown 2024-03-25T19:51:57.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb31924a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:57.618 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb31924a640 1 -- 172.21.15.69:0/748419173 >> v1:172.21.15.69:6789/0 conn(0x7fb3140b4d60 legacy=0x7fb31413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb31924a640 1 -- 172.21.15.69:0/748419173 shutdown_connections 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb31924a640 1 -- 172.21.15.69:0/748419173 >> 172.21.15.69:0/748419173 conn(0x7fb3140617a0 msgr2=0x7fb314061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.616+0000 7fb31924a640 1 -- 172.21.15.69:0/748419173 shutdown_connections 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.617+0000 7fb31924a640 1 -- 172.21.15.69:0/748419173 wait complete. 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.617+0000 7fb31924a640 10 monclient: build_initial_monmap 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.617+0000 7fb31924a640 10 monclient: monmap: 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:51:57.619 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:51:57.620 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:57.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.618+0000 7fb31924a640 1 librados: starting msgr at 2024-03-25T19:51:57.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.618+0000 7fb31924a640 1 librados: starting objecter 2024-03-25T19:51:57.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.618+0000 7fb31924a640 1 Processor -- start 2024-03-25T19:51:57.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.618+0000 7fb31924a640 1 -- start start 2024-03-25T19:51:57.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.618+0000 7fb31924a640 1 librados: setting wanted keys 2024-03-25T19:51:57.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.618+0000 7fb31924a640 1 librados: calling monclient init 2024-03-25T19:51:57.620 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.618+0000 7fb31924a640 10 monclient: init 2024-03-25T19:51:57.621 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:57.621 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:57.621 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 10 monclient(hunting): picked mon.noname-c con 0x7fb3140b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:51:57.621 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 10 monclient(hunting): picked mon.noname-a con 0x7fb3140b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:51:57.621 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 10 monclient(hunting): picked mon.noname-b con 0x7fb31414be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:51:57.621 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3141d7d80 con 0x7fb3140b4d60 2024-03-25T19:51:57.621 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3141d78a0 con 0x7fb31414be90 2024-03-25T19:51:57.621 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3141d7a30 con 0x7fb3140b1730 2024-03-25T19:51:57.622 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:57.622 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.619+0000 7fb31924a640 10 monclient(hunting): authenticate will time out at 5529.888149s 2024-03-25T19:51:57.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.642+0000 7fb31259c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb3140b4d60 0x7fb314143900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33656/0 (socket says 172.21.15.69:33656) 2024-03-25T19:51:57.642 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.642+0000 7fb31259c640 1 -- 172.21.15.69:0/85785046 learned_addr learned my addr 172.21.15.69:0/85785046 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:51:57.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.643+0000 7fb2fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb3140b4d60 2024-03-25T19:51:57.643 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.643+0000 7fb2fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.651+0000 7fb2fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb31414be90 2024-03-25T19:51:57.651 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.651+0000 7fb2fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.656 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.656+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892664838 0 0) 0x7fb3141d7d80 con 0x7fb3140b4d60 2024-03-25T19:51:57.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.656+0000 7fb2fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.656+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.656+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.656+0000 7fb2fb7fe640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:51:57.657 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.656+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb300004eb0 con 0x7fb3140b4d60 2024-03-25T19:51:57.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.662+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167400454 0 0) 0x7fb3141d78a0 con 0x7fb31414be90 2024-03-25T19:51:57.662 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.662+0000 7fb2fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.662+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.662+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.662+0000 7fb2fb7fe640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.662+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3141d7d80 con 0x7fb31414be90 2024-03-25T19:51:57.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.667+0000 7fb2fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb3140b1730 2024-03-25T19:51:57.667 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.667+0000 7fb2fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.678+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4256854385 0 0) 0x7fb3141d7d80 con 0x7fb31414be90 2024-03-25T19:51:57.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.678+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3141d78a0 con 0x7fb31414be90 2024-03-25T19:51:57.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.678+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb30c001b90 con 0x7fb31414be90 2024-03-25T19:51:57.678 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.678+0000 7fb2fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.678+0000 7fb2fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.678+0000 7fb2fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:57.679 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.678+0000 7fb2fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:57.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.680+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909320689 0 0) 0x7fb3141d7a30 con 0x7fb3140b1730 2024-03-25T19:51:57.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.680+0000 7fb2fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.680+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.680+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.680+0000 7fb2fb7fe640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:51:57.680 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.680+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3141d7d80 con 0x7fb3140b1730 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.686+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4258255213 0 0) 0x7fb300004eb0 con 0x7fb3140b4d60 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.686+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3141d7a30 con 0x7fb3140b4d60 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.686+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb3080018b0 con 0x7fb3140b4d60 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.686+0000 7fb2fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.686+0000 7fb2fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.686+0000 7fb2fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.686 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.687 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:57.687 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:57.687 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:57.687 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:57.687 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:57.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2904397892 0 0) 0x7fb3141d78a0 con 0x7fb31414be90 2024-03-25T19:51:57.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 >> v1:172.21.15.69:6790/0 conn(0x7fb3140b1730 legacy=0x7fb3141d8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 >> v1:172.21.15.69:6789/0 conn(0x7fb3140b4d60 legacy=0x7fb314143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:57.688 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 1 monclient: found mon.b 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb314136200 con 0x7fb31414be90 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb2fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 10 monclient: _renew_subs 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3141364d0 con 0x7fb31414be90 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 10 monclient: _renew_subs 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb314130040 con 0x7fb31414be90 2024-03-25T19:51:57.689 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.687+0000 7fb31924a640 1 librados: init done 2024-03-25T19:51:57.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.688+0000 7fb31924a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.688+0000 7fb31924a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.688+0000 7fb31924a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:57.690 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.688+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3141d7bc0 con 0x7fb31414be90 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.986+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb30c002040 con 0x7fb31414be90 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.986+0000 7fb2fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.986+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fb30c003990 con 0x7fb31414be90 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.986+0000 7fb2fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.986+0000 7fb2fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.986+0000 7fb2fb7fe640 10 monclient: dump: 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:51:57.986 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.987 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:51:57.987 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.987 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:51:57.987 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:51:57.987 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:51:57.987 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:51:57.987 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:51:57.987 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.986+0000 7fb31159a640 4 set_mon_vals no callback set 2024-03-25T19:51:57.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.986+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fb30c003af0 con 0x7fb31414be90 2024-03-25T19:51:57.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.987+0000 7fb312d9d640 10 monclient: get_auth_request con 0x7fb30006df70 auth_method 0 2024-03-25T19:51:57.988 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.987+0000 7fb2fb7fe640 10 client.4711.objecter ms_handle_connect 0x7fb30006df70 2024-03-25T19:51:57.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.992+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7fb30c053650 con 0x7fb31414be90 2024-03-25T19:51:57.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.992+0000 7fb2fb7fe640 10 client.4711.objecter ms_dispatch 0x7fb314000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:57.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.992+0000 7fb2fb7fe640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:57.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.992+0000 7fb2fb7fe640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:57.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.992+0000 7fb2fb7fe640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T19:51:57.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.992+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb30c04ef90 con 0x7fb31414be90 2024-03-25T19:51:57.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.992+0000 7fb2fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.992 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:57.992+0000 7fb2fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:58.242 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.242+0000 7fb31924a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.242+0000 7fb31924a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.242+0000 7fb31924a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T19:51:58.243 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.242+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb314138340 con 0x7fb31414be90 2024-03-25T19:51:58.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.277+0000 7fb2fb7fe640 1 -- 172.21.15.69:0/85785046 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7fb30c051090 con 0x7fb31414be90 2024-03-25T19:51:58.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.278+0000 7fb2fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.278 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.278+0000 7fb2fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:58.278 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:51:58.278 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 10 librados: watch_flush enter 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 10 librados: watch_flush exit 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 10 client.4711.objecter successfully canceled tick 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 >> v1:172.21.15.169:6800/3072548259 conn(0x7fb30006df70 legacy=0x7fb300070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 10 monclient: shutdown 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 >> v1:172.21.15.169:6789/0 conn(0x7fb31414be90 legacy=0x7fb3141369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 shutdown_connections 2024-03-25T19:51:58.281 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 >> 172.21.15.69:0/85785046 conn(0x7fb3140617a0 msgr2=0x7fb3140b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:58.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 shutdown_connections 2024-03-25T19:51:58.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.280+0000 7fb31924a640 1 -- 172.21.15.69:0/85785046 wait complete. 2024-03-25T19:51:58.282 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:51:58.281+0000 7fb31924a640 1 librados: shutdown 2024-03-25T19:51:58.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:05.100 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:05.214 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.214+0000 7fce2e625640 10 monclient: get_monmap_and_config 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.214+0000 7fce2e625640 10 monclient: build_initial_monmap 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.214+0000 7fce2e625640 10 monclient: monmap: 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:52:05.215962+0000 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:52:05.215962+0000 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:52:05.215 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.215+0000 7fce2e625640 1 Processor -- start 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.215+0000 7fce2e625640 1 -- start start 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.215+0000 7fce2e625640 10 monclient: init 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 10 monclient(hunting): picked mon.noname-b con 0x7fce280a98f0 addr v1:172.21.15.69:6790/0 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 10 monclient(hunting): picked mon.noname-a con 0x7fce2813f710 addr v1:172.21.15.69:6789/0 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 10 monclient(hunting): picked mon.noname-c con 0x7fce281430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce28142520 con 0x7fce2813f710 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce280b28a0 con 0x7fce281430c0 2024-03-25T19:52:05.217 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce280b2a30 con 0x7fce280a98f0 2024-03-25T19:52:05.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:05.218 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.216+0000 7fce2e625640 10 monclient(hunting): authenticate will time out at 5537.485440s 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.232+0000 7fce277fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fce2813f710 0x7fce281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46746/0 (socket says 172.21.15.69:46746) 2024-03-25T19:52:05.233 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.232+0000 7fce277fe640 1 -- 172.21.15.69:0/2277836051 learned_addr learned my addr 172.21.15.69:0/2277836051 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:52:05.248 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.248+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405144644 0 0) 0x7fce28142520 con 0x7fce2813f710 2024-03-25T19:52:05.248 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.248+0000 7fce26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.248 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.248+0000 7fce26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.248+0000 7fce26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.248+0000 7fce26ffd640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:52:05.249 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.248+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce10003520 con 0x7fce2813f710 2024-03-25T19:52:05.260 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.260+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048020127 0 0) 0x7fce280b2a30 con 0x7fce280a98f0 2024-03-25T19:52:05.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.260+0000 7fce26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.260+0000 7fce26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.260+0000 7fce26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.260+0000 7fce26ffd640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:52:05.261 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.260+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce28142520 con 0x7fce280a98f0 2024-03-25T19:52:05.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.270+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616558689 0 0) 0x7fce280b28a0 con 0x7fce281430c0 2024-03-25T19:52:05.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.270+0000 7fce26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.270+0000 7fce26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.270+0000 7fce26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.270+0000 7fce26ffd640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:52:05.270 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.270+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce280b2a30 con 0x7fce281430c0 2024-03-25T19:52:05.273 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.272+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 <== mon.2 v1:172.21.15.69:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3744377890 0 0) 0x7fce28142520 con 0x7fce280a98f0 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 >> v1:172.21.15.169:6789/0 conn(0x7fce281430c0 legacy=0x7fce2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 >> v1:172.21.15.69:6789/0 conn(0x7fce2813f710 legacy=0x7fce281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 1 monclient: found mon.noname-b 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.69:6790/0 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 --> v1:172.21.15.69:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce280b44c0 con 0x7fce280a98f0 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 10 monclient: _finish_auth 0 2024-03-25T19:52:05.274 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 <== mon.2 v1:172.21.15.69:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce18001670 con 0x7fce280a98f0 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce26ffd640 10 monclient: dump: 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:05.275 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:05.276 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce2e625640 5 monclient: authenticate success, global_id 4712 2024-03-25T19:52:05.276 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.273+0000 7fce2e625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:05.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.283+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 <== mon.2 v1:172.21.15.69:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce18001890 con 0x7fce280a98f0 2024-03-25T19:52:05.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.283+0000 7fce26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:05.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.283+0000 7fce26ffd640 1 -- 172.21.15.69:0/2277836051 <== mon.2 v1:172.21.15.69:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce18001c20 con 0x7fce280a98f0 2024-03-25T19:52:05.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.283+0000 7fce26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.283+0000 7fce26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:05.283 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.283+0000 7fce26ffd640 10 monclient: dump: 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.283+0000 7fce2e625640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:05.284 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.283+0000 7fce2e625640 4 set_mon_vals no callback set 2024-03-25T19:52:05.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.284+0000 7fce2e625640 10 monclient: shutdown 2024-03-25T19:52:05.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.284+0000 7fce2e625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:05.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.284+0000 7fce2e625640 1 -- 172.21.15.69:0/2277836051 >> v1:172.21.15.69:6790/0 conn(0x7fce280a98f0 legacy=0x7fce280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.284+0000 7fce2e625640 1 -- 172.21.15.69:0/2277836051 shutdown_connections 2024-03-25T19:52:05.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.284+0000 7fce2e625640 1 -- 172.21.15.69:0/2277836051 >> 172.21.15.69:0/2277836051 conn(0x7fce280b7ea0 msgr2=0x7fce280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:05.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.284+0000 7fce2e625640 1 -- 172.21.15.69:0/2277836051 shutdown_connections 2024-03-25T19:52:05.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.284+0000 7fce2e625640 1 -- 172.21.15.69:0/2277836051 wait complete. 2024-03-25T19:52:05.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.285+0000 7fce2e625640 10 monclient: build_initial_monmap 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.285+0000 7fce2e625640 10 monclient: monmap: 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.285+0000 7fce2e625640 1 librados: starting msgr at 2024-03-25T19:52:05.288 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.285+0000 7fce2e625640 1 librados: starting objecter 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.285+0000 7fce2e625640 1 Processor -- start 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.286+0000 7fce2e625640 1 -- start start 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.286+0000 7fce2e625640 1 librados: setting wanted keys 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.286+0000 7fce2e625640 1 librados: calling monclient init 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.286+0000 7fce2e625640 10 monclient: init 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.286+0000 7fce2e625640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 10 monclient(hunting): picked mon.noname-a con 0x7fce280a98f0 addr v1:172.21.15.69:6789/0 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 10 monclient(hunting): picked mon.noname-c con 0x7fce2813f710 addr v1:172.21.15.69:6790/0 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 10 monclient(hunting): picked mon.noname-b con 0x7fce281430c0 addr v1:172.21.15.169:6789/0 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce281dc070 con 0x7fce280a98f0 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce28138060 con 0x7fce281430c0 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce28136e90 con 0x7fce2813f710 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:05.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.287+0000 7fce2e625640 10 monclient(hunting): authenticate will time out at 5537.555810s 2024-03-25T19:52:05.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.295+0000 7fce27fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fce280a98f0 0x7fce28137950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46778/0 (socket says 172.21.15.69:46778) 2024-03-25T19:52:05.295 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.295+0000 7fce27fff640 1 -- 172.21.15.69:0/944343770 learned_addr learned my addr 172.21.15.69:0/944343770 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:52:05.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.296+0000 7fce24ff9640 10 client.?.objecter ms_handle_connect 0x7fce280a98f0 2024-03-25T19:52:05.296 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.296+0000 7fce24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.300+0000 7fce24ff9640 10 client.?.objecter ms_handle_connect 0x7fce281430c0 2024-03-25T19:52:05.300 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.300+0000 7fce24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422370132 0 0) 0x7fce281dc070 con 0x7fce280a98f0 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce14004eb0 con 0x7fce280a98f0 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760890189 0 0) 0x7fce28138060 con 0x7fce281430c0 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:52:05.312 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.311+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce281dc070 con 0x7fce281430c0 2024-03-25T19:52:05.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.314+0000 7fce24ff9640 10 client.?.objecter ms_handle_connect 0x7fce2813f710 2024-03-25T19:52:05.314 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.314+0000 7fce24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.329+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716411680 0 0) 0x7fce14004eb0 con 0x7fce280a98f0 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.330+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce28138060 con 0x7fce280a98f0 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.330+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce18001c60 con 0x7fce280a98f0 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.330+0000 7fce24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.330+0000 7fce24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.330+0000 7fce24ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:05.330 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:05.331 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:05.331 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:05.331 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:05.331 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.330+0000 7fce24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:05.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.334+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3011350723 0 0) 0x7fce281dc070 con 0x7fce281430c0 2024-03-25T19:52:05.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.334+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce14004eb0 con 0x7fce281430c0 2024-03-25T19:52:05.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.334+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce20001bd0 con 0x7fce281430c0 2024-03-25T19:52:05.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.334+0000 7fce24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.334+0000 7fce24ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:05.334 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.334+0000 7fce24ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:05.334 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:05.335 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:05.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.343+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788354411 0 0) 0x7fce28136e90 con 0x7fce2813f710 2024-03-25T19:52:05.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.343+0000 7fce24ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.343 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.343+0000 7fce24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.343+0000 7fce24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.343+0000 7fce24ff9640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:52:05.344 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.343+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce281dc070 con 0x7fce2813f710 2024-03-25T19:52:05.352 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1133427540 0 0) 0x7fce28138060 con 0x7fce280a98f0 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 >> v1:172.21.15.69:6790/0 conn(0x7fce2813f710 legacy=0x7fce281db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 >> v1:172.21.15.169:6789/0 conn(0x7fce281430c0 legacy=0x7fce281db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 1 monclient: found mon.a 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28137170 con 0x7fce280a98f0 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 10 monclient: _finish_auth 0 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 5 monclient: authenticate success, global_id 4743 2024-03-25T19:52:05.353 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 10 monclient: _renew_subs 2024-03-25T19:52:05.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:05.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2813cbb0 con 0x7fce280a98f0 2024-03-25T19:52:05.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 10 client.4743.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:05.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 10 monclient: _renew_subs 2024-03-25T19:52:05.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:05.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce281361c0 con 0x7fce280a98f0 2024-03-25T19:52:05.354 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.352+0000 7fce2e625640 1 librados: init done 2024-03-25T19:52:05.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.353+0000 7fce2e625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.353+0000 7fce2e625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.353+0000 7fce2e625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:05.355 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.353+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce280b3ef0 con 0x7fce280a98f0 2024-03-25T19:52:05.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.658+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce180038a0 con 0x7fce280a98f0 2024-03-25T19:52:05.658 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.658+0000 7fce24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.658+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7fce18003cb0 con 0x7fce280a98f0 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.658+0000 7fce24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.658+0000 7fce24ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.658+0000 7fce24ff9640 10 monclient: dump: 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:05.659 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:05.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.658+0000 7fce267fc640 4 set_mon_vals no callback set 2024-03-25T19:52:05.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.659+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7fce18003ed0 con 0x7fce280a98f0 2024-03-25T19:52:05.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.659+0000 7fce277fe640 10 monclient: get_auth_request con 0x7fce1406df70 auth_method 0 2024-03-25T19:52:05.660 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.660+0000 7fce24ff9640 10 client.4743.objecter ms_handle_connect 0x7fce1406df70 2024-03-25T19:52:05.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.664+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7fce180549a0 con 0x7fce280a98f0 2024-03-25T19:52:05.664 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.664+0000 7fce24ff9640 10 client.4743.objecter ms_dispatch 0x7fce28000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:05.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.664+0000 7fce24ff9640 3 client.4743.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:05.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.664+0000 7fce24ff9640 3 client.4743.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:05.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.664+0000 7fce24ff9640 20 client.4743.objecter dump_active .. 0 homeless 2024-03-25T19:52:05.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.664+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce180504b0 con 0x7fce280a98f0 2024-03-25T19:52:05.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.664+0000 7fce24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.665 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.664+0000 7fce24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:05.894 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.894+0000 7fce2e625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:05.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.894+0000 7fce2e625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:05.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.894+0000 7fce2e625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:05.895 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.894+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce280627b0 con 0x7fce280a98f0 2024-03-25T19:52:05.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.947+0000 7fce24ff9640 1 -- 172.21.15.69:0/944343770 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7fce180523e0 con 0x7fce280a98f0 2024-03-25T19:52:05.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.947+0000 7fce24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:05.947 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.947+0000 7fce24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:05.947 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:52:05.947 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 10 librados: watch_flush enter 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 10 librados: watch_flush exit 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 20 client.4743.objecter shutdown clearing up homeless session... 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 10 client.4743.objecter successfully canceled tick 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 >> v1:172.21.15.169:6800/3072548259 conn(0x7fce1406df70 legacy=0x7fce14070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 10 monclient: shutdown 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 >> v1:172.21.15.69:6789/0 conn(0x7fce280a98f0 legacy=0x7fce28137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 shutdown_connections 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 >> 172.21.15.69:0/944343770 conn(0x7fce280b7ea0 msgr2=0x7fce28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 shutdown_connections 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 1 -- 172.21.15.69:0/944343770 wait complete. 2024-03-25T19:52:05.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:05.949+0000 7fce2e625640 1 librados: shutdown 2024-03-25T19:52:05.962 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:12.768 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:12.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.867+0000 7f810a10d640 10 monclient: get_monmap_and_config 2024-03-25T19:52:12.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.867+0000 7f810a10d640 10 monclient: build_initial_monmap 2024-03-25T19:52:12.867 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.867+0000 7f810a10d640 10 monclient: monmap: 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:52:12.868828+0000 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:52:12.868828+0000 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-b 2024-03-25T19:52:12.868 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:12.869 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.868+0000 7f810a10d640 1 Processor -- start 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.868+0000 7f810a10d640 1 -- start start 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.868+0000 7f810a10d640 10 monclient: init 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.868+0000 7f810a10d640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.868+0000 7f810a10d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.869+0000 7f810a10d640 10 monclient(hunting): picked mon.noname-b con 0x7f81040b4d60 addr v1:172.21.15.69:6790/0 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.869+0000 7f810a10d640 10 monclient(hunting): picked mon.noname-c con 0x7f81040b1730 addr v1:172.21.15.169:6789/0 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.869+0000 7f810a10d640 10 monclient(hunting): picked mon.noname-a con 0x7f810414be90 addr v1:172.21.15.69:6789/0 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.869+0000 7f810a10d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81040af940 con 0x7f810414be90 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.869+0000 7f810a10d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81040b0a70 con 0x7f81040b1730 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.869+0000 7f810a10d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81040b0d50 con 0x7f81040b4d60 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.869+0000 7f810a10d640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:12.870 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.869+0000 7f810a10d640 10 monclient(hunting): authenticate will time out at 5545.137711s 2024-03-25T19:52:12.878 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.879+0000 7f8102ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f81040b1730 0x7f81040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.69:49048/0 (socket says 172.21.15.69:49048) 2024-03-25T19:52:12.879 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.879+0000 7f8102ffd640 1 -- 172.21.15.69:0/3923317393 learned_addr learned my addr 172.21.15.69:0/3923317393 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:52:12.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.899+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832450933 0 0) 0x7f81040b0a70 con 0x7f81040b1730 2024-03-25T19:52:12.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.899+0000 7f81027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:12.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.899+0000 7f81027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:12.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.899+0000 7f81027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:12.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.899+0000 7f81027fc640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:52:12.899 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.899+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80ec003520 con 0x7f81040b1730 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1299515601 0 0) 0x7f80ec003520 con 0x7f81040b1730 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 >> v1:172.21.15.69:6790/0 conn(0x7f81040b4d60 legacy=0x7f810413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 >> v1:172.21.15.69:6789/0 conn(0x7f810414be90 legacy=0x7f810414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 10 monclient: _finish_hunting 0 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 1 monclient: found mon.noname-c 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81040b1030 con 0x7f81040b1730 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 10 monclient: _finish_auth 0 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 10 monclient: _check_auth_tickets 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f80f8001690 con 0x7f81040b1730 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f810a10d640 5 monclient: authenticate success, global_id 4720 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f810a10d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f81027fc640 10 monclient: dump: 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:12.920 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:12.920 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:12.920 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:12.920 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:12.920 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:12.920 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:12.920 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.918+0000 7f810a10d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.929+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f80017f0 con 0x7f81040b1730 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.929+0000 7f81027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.929+0000 7f81027fc640 1 -- 172.21.15.69:0/3923317393 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f80f8001c00 con 0x7f81040b1730 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.929+0000 7f81027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.929+0000 7f81027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.929+0000 7f81027fc640 10 monclient: dump: 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:12.929 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.929+0000 7f810a10d640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:12.930 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.929+0000 7f810a10d640 4 set_mon_vals no callback set 2024-03-25T19:52:12.932 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.930+0000 7f810a10d640 10 monclient: shutdown 2024-03-25T19:52:12.932 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.930+0000 7f810a10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.930+0000 7f810a10d640 1 -- 172.21.15.69:0/3923317393 >> v1:172.21.15.169:6789/0 conn(0x7f81040b1730 legacy=0x7f81040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.930+0000 7f810a10d640 1 -- 172.21.15.69:0/3923317393 shutdown_connections 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.930+0000 7f810a10d640 1 -- 172.21.15.69:0/3923317393 >> 172.21.15.69:0/3923317393 conn(0x7f81040617a0 msgr2=0x7f8104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.930+0000 7f810a10d640 1 -- 172.21.15.69:0/3923317393 shutdown_connections 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.930+0000 7f810a10d640 1 -- 172.21.15.69:0/3923317393 wait complete. 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 10 monclient: build_initial_monmap 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 10 monclient: monmap: 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-03-25T19:52:12.933 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.noname-c 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 1 librados: starting msgr at 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 1 librados: starting objecter 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 1 Processor -- start 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 1 -- start start 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 1 librados: setting wanted keys 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 1 librados: calling monclient init 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.931+0000 7f810a10d640 10 monclient: init 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 10 monclient(hunting): picked mon.noname-c con 0x7f81040b1730 addr v1:172.21.15.69:6790/0 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 10 monclient(hunting): picked mon.noname-a con 0x7f81040b4d60 addr v1:172.21.15.69:6789/0 2024-03-25T19:52:12.934 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 10 monclient(hunting): picked mon.noname-b con 0x7f810414be90 addr v1:172.21.15.169:6789/0 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8104142050 con 0x7f81040b4d60 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81041dac50 con 0x7f810414be90 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 1 -- --> v1:172.21.15.69:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8104140d00 con 0x7f81040b1730 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f810a10d640 10 monclient(hunting): authenticate will time out at 5545.201089s 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f8102ffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f81040b4d60 0x7f81041db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42122/0 (socket says 172.21.15.69:42122) 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.932+0000 7f8102ffd640 1 -- 172.21.15.69:0/1597467852 learned_addr learned my addr 172.21.15.69:0/1597467852 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.933+0000 7f80ebfff640 10 client.?.objecter ms_handle_connect 0x7f81040b4d60 2024-03-25T19:52:12.935 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.933+0000 7f80ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:12.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.941+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378493802 0 0) 0x7f8104142050 con 0x7f81040b4d60 2024-03-25T19:52:12.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.941+0000 7f80ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:12.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.941+0000 7f80ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:12.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.941+0000 7f80ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:12.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.941+0000 7f80ebfff640 10 monclient(hunting): my global_id is 4749 2024-03-25T19:52:12.941 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.941+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f0004eb0 con 0x7f81040b4d60 2024-03-25T19:52:12.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.944+0000 7f80ebfff640 10 client.?.objecter ms_handle_connect 0x7f81040b1730 2024-03-25T19:52:12.944 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.944+0000 7f80ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:12.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 10 client.?.objecter ms_handle_connect 0x7f810414be90 2024-03-25T19:52:12.949 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:12.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3246874271 0 0) 0x7f80f0004eb0 con 0x7f81040b4d60 2024-03-25T19:52:12.950 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 --> v1:172.21.15.69:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8104142050 con 0x7f81040b4d60 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f80f8001c00 con 0x7f81040b4d60 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 10 monclient(hunting): dump: 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:12.951 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.950+0000 7f80ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:12.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.953+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.2 v1:172.21.15.69:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223848741 0 0) 0x7f8104140d00 con 0x7f81040b1730 2024-03-25T19:52:12.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.953+0000 7f80ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:12.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.953+0000 7f80ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:12.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.953+0000 7f80ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:12.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.953+0000 7f80ebfff640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:52:12.953 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.953+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 --> v1:172.21.15.69:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f0004eb0 con 0x7f81040b1730 2024-03-25T19:52:12.959 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 527722810 0 0) 0x7f8104142050 con 0x7f81040b4d60 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 >> v1:172.21.15.69:6790/0 conn(0x7f81040b1730 legacy=0x7f81041405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 >> v1:172.21.15.169:6789/0 conn(0x7f810414be90 legacy=0x7f81041e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 1 monclient: found mon.a 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81041423e0 con 0x7f81040b4d60 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 10 monclient: _finish_auth 0 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f80ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 5 monclient: authenticate success, global_id 4749 2024-03-25T19:52:12.960 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 10 monclient: _renew_subs 2024-03-25T19:52:12.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:12.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 --> v1:172.21.15.69:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810413b470 con 0x7f81040b4d60 2024-03-25T19:52:12.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 10 client.4749.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:12.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 10 monclient: _renew_subs 2024-03-25T19:52:12.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:12.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 --> v1:172.21.15.69:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81041dba40 con 0x7f81040b4d60 2024-03-25T19:52:12.961 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.959+0000 7f810a10d640 1 librados: init done 2024-03-25T19:52:12.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.960+0000 7f810a10d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:12.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.960+0000 7f810a10d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:12.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.960+0000 7f810a10d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:12.962 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:12.960+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81041401f0 con 0x7f81040b4d60 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.287+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f8004a90 con 0x7f81040b4d60 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.287+0000 7f80ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.287+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1246691614 0 0) 0x7f80f8004e60 con 0x7f81040b4d60 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.287+0000 7f80ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.287+0000 7f80ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.287+0000 7f80ebfff640 10 monclient: dump: 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:fsid 0b4416d3-973b-4b34-9bff-820f05cd8197 2024-03-25T19:52:13.287 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:13.288 INFO:teuthology.misc.health.smithi069.stderr:created 2024-03-25T19:39:48.802568+0000 2024-03-25T19:52:13.288 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.288 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 3 2024-03-25T19:52:13.288 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.a 2024-03-25T19:52:13.288 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T19:52:13.288 INFO:teuthology.misc.health.smithi069.stderr:2: v1:172.21.15.69:6790/0 mon.c 2024-03-25T19:52:13.288 INFO:teuthology.misc.health.smithi069.stderr: 2024-03-25T19:52:13.288 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.287+0000 7f8101ffb640 4 set_mon_vals no callback set 2024-03-25T19:52:13.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.287+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4081362119 0 0) 0x7f80f8005080 con 0x7f81040b4d60 2024-03-25T19:52:13.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.288+0000 7f81037fe640 10 monclient: get_auth_request con 0x7f80f006ddf0 auth_method 0 2024-03-25T19:52:13.289 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.288+0000 7f80ebfff640 10 client.4749.objecter ms_handle_connect 0x7f80f006ddf0 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.293+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3578106889 0 0) 0x7f80f8054990 con 0x7f81040b4d60 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.293+0000 7f80ebfff640 10 client.4749.objecter ms_dispatch 0x7f8104000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.293+0000 7f80ebfff640 3 client.4749.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.293+0000 7f80ebfff640 3 client.4749.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.293+0000 7f80ebfff640 20 client.4749.objecter dump_active .. 0 homeless 2024-03-25T19:52:13.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.293+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80f80504a0 con 0x7f81040b4d60 2024-03-25T19:52:13.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.293+0000 7f80ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:13.294 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.293+0000 7f80ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:13.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.546+0000 7f810a10d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.546+0000 7f810a10d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.546+0000 7f810a10d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.69:6789/0 2024-03-25T19:52:13.547 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.546+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 --> v1:172.21.15.69:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8104140e90 con 0x7f81040b4d60 2024-03-25T19:52:13.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.570+0000 7f80ebfff640 1 -- 172.21.15.69:0/1597467852 <== mon.0 v1:172.21.15.69:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2085260049) 0x7f80f80523d0 con 0x7f81040b4d60 2024-03-25T19:52:13.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.570+0000 7f80ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.570 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.570+0000 7f80ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:13.570 INFO:teuthology.misc.health.smithi069.stdout: 2024-03-25T19:52:13.570 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":11,"fsid":"0b4416d3-973b-4b34-9bff-820f05cd8197","created":"2024-03-25T19:40:44.346598+0000","modified":"2024-03-25T19:50:48.697217+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:48.697217+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7c9ae774-73d1-442a-8d13-58702d808d34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"68679d85-c21b-44b4-9f57-a4af14c984d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a86395a9-8a87-4f9f-8589-2a4ec6c5a991","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dae35ad2-6d51-40b6-b4fc-8d377aee1f23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"07990f28-82fb-49d8-b33d-02afbd20c66a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df15ac59-21c6-41e8-8f03-eeb135940c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7d6304e-042b-4de5-a93d-983af03ad23a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"16becd40-0b67-4e42-b407-63f5b24cd430","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:13.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.572+0000 7f810a10d640 10 librados: watch_flush enter 2024-03-25T19:52:13.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.572+0000 7f810a10d640 10 librados: watch_flush exit 2024-03-25T19:52:13.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.572+0000 7f810a10d640 20 client.4749.objecter shutdown clearing up homeless session... 2024-03-25T19:52:13.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.572+0000 7f810a10d640 10 client.4749.objecter successfully canceled tick 2024-03-25T19:52:13.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 >> v1:172.21.15.169:6800/3072548259 conn(0x7f80f006ddf0 legacy=0x7f80f00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 10 monclient: shutdown 2024-03-25T19:52:13.573 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:13.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 >> v1:172.21.15.69:6789/0 conn(0x7f81040b4d60 legacy=0x7f81041db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 shutdown_connections 2024-03-25T19:52:13.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 >> 172.21.15.69:0/1597467852 conn(0x7f81040617a0 msgr2=0x7f81040b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:13.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 shutdown_connections 2024-03-25T19:52:13.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 1 -- 172.21.15.69:0/1597467852 wait complete. 2024-03-25T19:52:13.574 INFO:teuthology.misc.health.smithi069.stderr:2024-03-25T19:52:13.573+0000 7f810a10d640 1 librados: shutdown 2024-03-25T19:52:13.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:13.585 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:52:13.586 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:52:13.587 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:52:13.587 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:14.053 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:52.017 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:14.059 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:52.023 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:14.096 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:52.060 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:14.113 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:52.067 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:14.113 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:52.077 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:14.125 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:52.079 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:14.130 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:52.094 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:14.159 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:52.113 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:14.166 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:52.130 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:15.637 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:11:25.392 35694== 2024-03-25T19:52:15.637 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:11:25.392 35694== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:15.679 INFO:tasks.ceph.osd.0.smithi069.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:19.689 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:52:19.690 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:52:19.690 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:52:19.690 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:21.726 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:11:31.462 35695== 2024-03-25T19:52:21.726 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:11:31.462 35695== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:21.766 INFO:tasks.ceph.osd.1.smithi069.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:25.792 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:52:25.792 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:52:25.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:52:25.793 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:27.755 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:11:37.486 35696== 2024-03-25T19:52:27.755 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:11:37.486 35696== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:27.796 INFO:tasks.ceph.osd.2.smithi069.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:31.894 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:52:31.895 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:52:31.895 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:52:31.895 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:33.834 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:11:43.565 35697== 2024-03-25T19:52:33.834 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:11:43.565 35697== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:33.877 INFO:tasks.ceph.osd.3.smithi069.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:34.030 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:11.994 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:34.084 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:12.048 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:34.114 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:12.078 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:34.132 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:12.086 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:34.134 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:12.098 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:34.141 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:12.095 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:34.145 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:12.109 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:34.164 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:12.118 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:34.172 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:12.136 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:37.997 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:52:37.998 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:52:37.998 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:52:37.998 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:39.915 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:11:49.663 35590== 2024-03-25T19:52:39.915 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:11:49.663 35590== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:39.956 INFO:tasks.ceph.osd.4.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:44.100 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:52:44.100 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:52:44.100 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:52:44.100 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:45.927 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:11:55.649 35597== 2024-03-25T19:52:45.927 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:11:55.649 35597== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:45.960 INFO:tasks.ceph.osd.5.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:50.203 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:52:50.203 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:52:50.203 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:52:50.204 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:52.013 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:12:01.755 35591== 2024-03-25T19:52:52.013 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:12:01.755 35591== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:52.058 INFO:tasks.ceph.osd.6.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:54.138 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:32.102 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:54.142 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:32.106 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:54.157 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:32.121 35381== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:54.184 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:32.138 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:54.187 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:32.141 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:54.188 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:32.152 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:54.192 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:32.156 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:54.202 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:32.157 35047== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:54.211 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:32.175 35382== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:56.306 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:52:56.306 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:52:56.307 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:52:56.307 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:57.779 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:12:07.521 35592== 2024-03-25T19:52:57.779 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:12:07.521 35592== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:57.809 INFO:tasks.ceph.osd.7.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:02.408 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:53:02.409 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:53:02.409 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:53:02.409 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:53:02.409 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:02.437 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:53:02.437 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:53:02.437 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:02.467 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:53:02.467 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:53:02.467 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:53:02.467 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:02.493 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-03-25T19:53:02.482+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35348) UID: 0 2024-03-25T19:53:02.494 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-03-25T19:53:02.485+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:58:01.540 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:58:01.541 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:58:01.592 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:58:01.592 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:58:01.663 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:58:01.664 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:58:01.738 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:58:01.738 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:58:01.790 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:58:01.790 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:58:01.856 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:58:01.857 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:58:02.000 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:58:02.000 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:58:02.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:58:02.055 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:58:02.112 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:58:02.112 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:58:02.171 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:58:02.173 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/data/mon.a.tgz 2024-03-25T19:58:02.173 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-03-25T19:58:02.187 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.lCrTqfqOCf 2024-03-25T19:58:02.187 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.lCrTqfqOCf 2024-03-25T19:58:02.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.lCrTqfqOCf 2024-03-25T19:58:02.371 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.lCrTqfqOCf is 133KB 2024-03-25T19:58:02.422 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.lCrTqfqOCf 2024-03-25T19:58:02.437 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/data/mon.c.tgz 2024-03-25T19:58:02.437 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-03-25T19:58:02.490 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.VdDuNDIXeQ 2024-03-25T19:58:02.490 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.VdDuNDIXeQ 2024-03-25T19:58:02.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.VdDuNDIXeQ 2024-03-25T19:58:02.666 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.VdDuNDIXeQ is 165KB 2024-03-25T19:58:02.717 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.VdDuNDIXeQ 2024-03-25T19:58:02.731 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/data/mon.b.tgz 2024-03-25T19:58:02.731 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-25T19:58:02.749 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.cRlsuMwxXk 2024-03-25T19:58:02.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cRlsuMwxXk 2024-03-25T19:58:02.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.cRlsuMwxXk 2024-03-25T19:58:02.956 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.cRlsuMwxXk is 156KB 2024-03-25T19:58:03.009 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.cRlsuMwxXk 2024-03-25T19:58:03.024 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:58:03.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:58:03.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:58:03.164 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:58:03.210 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:58:03.407 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:58:03.483 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:58:03.676 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:58:03.678 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/remote/smithi069/crash 2024-03-25T19:58:03.679 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:58:03.706 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/remote/smithi169/crash 2024-03-25T19:58:03.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:58:03.734 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:58:03.734 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:58:03.751 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:58:03.824 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:58:03.824 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35360.log 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35280.log 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35320.log 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35280.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35360.log.gz 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35320.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35280.log.gz 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35320.log.gz 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:58:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr: 55.9% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35450.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35491.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35450.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35491.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35450.log.gz 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.audit.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35491.log.gz 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr: 87.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35240.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35400.log 2024-03-25T19:58:03.826 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35240.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:58:03.827 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35400.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:58:03.827 INFO:teuthology.orchestra.run.smithi169.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35240.log.gz 2024-03-25T19:58:03.827 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mgr.x.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:58:03.827 INFO:teuthology.orchestra.run.smithi169.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35400.log.gz 2024-03-25T19:58:03.827 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:58:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42450.log 2024-03-25T19:58:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40144.log 2024-03-25T19:58:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42450.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:58:03.831 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42528.log 2024-03-25T19:58:03.831 INFO:teuthology.orchestra.run.smithi069.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42450.log.gz 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39676.log 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40144.log.gz 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42528.log: /var/log/ceph/ceph-client.admin.39676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42528.log.gz 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39676.log.gz 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42294.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40417.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/mon.c.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr: 55.8% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:58:03.832 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.2.log: /var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr: 55.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41356.log 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40888.log 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41122.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41044.log 2024-03-25T19:58:03.833 INFO:teuthology.orchestra.run.smithi069.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40888.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41356.log.gz 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42489.log 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39283.log 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41044.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40888.log.gz 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41902.log 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41044.log.gz 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37735.log 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42489.log.gz 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40339.log 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41902.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39283.log.gz 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40771.log 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40339.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41902.log.gz 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41005.log 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.37735.log.gz 2024-03-25T19:58:03.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40693.log 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40339.log.gz 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35390.log 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40771.log.gz 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.35390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40693.log.gz 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41395.log 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39024.log 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35390.log.gz 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41395.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42372.log 2024-03-25T19:58:03.835 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42411.log 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39024.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41395.log.gz 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42372.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42411.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39024.log.gz 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42372.log.gz 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42411.log.gz 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39636.log: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35693.log 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42019.log 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.35693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42099.log 2024-03-25T19:58:03.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42019.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.35693.log.gz 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41707.log 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42099.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42019.log.gz 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40732.log 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42099.log.gz 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40732.log: 86.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.41707.log.gz 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42060.log 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40732.log.gz 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35720.log 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42060.log.gz 2024-03-25T19:58:03.837 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40183.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40027.log 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.35720.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39244.log 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40027.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.35720.log.gz 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39244.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40027.log.gz 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41551.log 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39244.log.gz 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40222.log 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40222.log: gzip -5 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41551.log.gz 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41980.log 2024-03-25T19:58:03.838 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39480.log: gzip 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40222.log.gz 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41668.log 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41980.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41668.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41980.log.gz 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41668.log.gz 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39832.log 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39832.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40927.log 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39519.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log 2024-03-25T19:58:03.839 INFO:teuthology.orchestra.run.smithi069.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42138.log 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40927.log.gz 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39754.log 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41434.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40654.log 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39754.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42138.log.gz 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39754.log.gz 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41863.log 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41317.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40654.log.gz 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40495.log 2024-03-25T19:58:03.840 INFO:teuthology.orchestra.run.smithi069.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41512.log 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41863.log.gz 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40378.log 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40849.log 86.6% 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40495.log.gz 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40378.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41512.log.gz 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40849.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39988.log 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39441.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40849.log.gz 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39988.log: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42333.log 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz 2024-03-25T19:58:03.841 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-mgr.y.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41785.log 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39988.log.gz 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40810.log 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42333.log: /var/log/ceph/ceph-client.admin.41785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39715.log 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40810.log: 86.8% 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42333.log.gz 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41785.log.gz 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40810.log.gz 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41941.log 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39715.log: /var/log/ceph/ceph-client.admin.41941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41161.log 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42216.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39715.log.gz 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41941.log.gz 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz 2024-03-25T19:58:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42177.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41161.log: /var/log/ceph/ceph-client.admin.41200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41590.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42177.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41161.log.gz 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41629.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42177.log.gz 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41590.log: /var/log/ceph/ceph-client.admin.41629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41746.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41590.log.gz 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.0.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41629.log.gz 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36755.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40534.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41746.log: /var/log/ceph/ceph-client.admin.36755.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41746.log.gz 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36755.log.gz 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41824.log 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40534.log.gz 2024-03-25T19:58:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29523.log 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41824.log: 86.9% 92.5% -- replaced with /var/log/ceph/ceph-client.admin.41824.log.gz 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40966.log 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40966.log: /var/log/ceph/ceph.tmp-client.admin.29523.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29523.log.gz 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:gzip 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz -5 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35864.log 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40966.log.gz 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.35864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39597.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35864.log.gz 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42255.log 2024-03-25T19:58:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41278.log 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr: 86.4%/var/log/ceph/ceph-client.admin.41278.log: -- replaced with /var/log/ceph/ceph-client.admin.42255.log.gz 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41239.log 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41278.log.gz 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39793.log 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41239.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41239.log.gz 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39949.log 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40066.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39793.log.gz 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39871.log 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39949.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41473.log 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39949.log.gz 2024-03-25T19:58:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40456.log 2024-03-25T19:58:03.846 INFO:teuthology.orchestra.run.smithi069.stderr: 86.8%/var/log/ceph/ceph-client.admin.41473.log: -- replaced with /var/log/ceph/ceph-client.admin.39871.log.gz 2024-03-25T19:58:03.846 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:58:03.846 INFO:teuthology.orchestra.run.smithi069.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41473.log.gz 2024-03-25T19:58:03.846 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40456.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40456.log.gz 2024-03-25T19:58:03.887 INFO:teuthology.orchestra.run.smithi169.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:58:04.490 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:42.454 35381== 2024-03-25T19:58:04.491 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:42.454 35381== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:58:04.506 INFO:tasks.ceph.mon.a.smithi069.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:58:04.671 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:58:04.728 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:58:04.734 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.c.log: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:58:04.758 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:58:04.791 INFO:teuthology.orchestra.run.smithi069.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:58:04.970 INFO:teuthology.orchestra.run.smithi069.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:58:04.986 INFO:teuthology.orchestra.run.smithi069.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:58:05.017 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:58:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2024-03-25T19:58:08.940 INFO:teuthology.orchestra.run.smithi069.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:58:09.170 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2024-03-25T19:58:09.170 INFO:teuthology.orchestra.run.smithi169.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:58:09.172 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T19:58:09.172 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m5.379s 2024-03-25T19:58:09.172 INFO:teuthology.orchestra.run.smithi169.stderr:user 0m9.147s 2024-03-25T19:58:09.172 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m0.257s 2024-03-25T19:58:09.177 INFO:teuthology.orchestra.run.smithi069.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2024-03-25T19:58:09.241 INFO:teuthology.orchestra.run.smithi069.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:58:09.243 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-25T19:58:09.243 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m5.481s 2024-03-25T19:58:09.243 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m14.491s 2024-03-25T19:58:09.243 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m0.500s 2024-03-25T19:58:09.243 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:58:09.244 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/remote/smithi069/log 2024-03-25T19:58:09.245 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:58:11.725 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/remote/smithi169/log 2024-03-25T19:58:11.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:58:13.277 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:58:13.502 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ea0bcbf940754433adb7466407354ef4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:58:13.508 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:58:13.518 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:58:13.573 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:58:13.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:58:13.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:58:13.672 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:58:13.684 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:58:13.684 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:58:13.687 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:58:13.701 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:58:13.703 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:58:13.706 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:58:13.706 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-03-25T19:58:13.706 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -562us[ -550us] +/- 67ms 2024-03-25T19:58:13.707 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -132us[ -120us] +/- 70ms 2024-03-25T19:58:13.707 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +743us[ +743us] +/- 73ms 2024-03-25T19:58:13.707 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:58:13.708 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:58:13.709 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-25T19:58:13.709 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 -539us[ -503us] +/- 67ms 2024-03-25T19:58:13.709 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -127us[ -90us] +/- 70ms 2024-03-25T19:58:13.709 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +763us[ +799us] +/- 73ms 2024-03-25T19:58:13.709 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:58:13.709 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:58:13.721 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:58:13.723 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:58:13.748 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:58:13.819 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:58:13.889 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:58:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:13.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:58:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:13.952 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:58:13.964 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:58:14.020 INFO:teuthology.task.internal:Duration was 5574.213811 seconds 2024-03-25T19:58:14.020 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:58:14.046 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:58:14.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:58:14.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:58:14.092 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:58:14.093 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:58:14.404 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:58:14.404 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-03-25T19:58:14.404 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:58:14.424 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T19:58:14.425 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:58:14.446 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:58:14.447 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:58:14.467 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:58:14.515 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:58:14.516 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:58:14.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:58:14.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:58:14.668 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:58:14.668 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:58:14.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:58:14.723 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:58:14.734 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:14.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:14.761 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-03-25T19:58:14.789 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-03-25T19:58:14.834 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:14.849 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:14.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:58:14.899 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:58:14.900 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/remote/smithi069 2024-03-25T19:58:14.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:58:14.940 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324/remote/smithi169 2024-03-25T19:58:14.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:58:14.978 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:58:14.978 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:58:14.981 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:58:15.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:58:15.039 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:58:15.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:58:15.056 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:58:15.056 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:58:15.058 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:58:15.070 INFO:teuthology.orchestra.run.smithi069.stdout: 265309 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:58 /home/ubuntu/cephtest 2024-03-25T19:58:15.081 INFO:teuthology.orchestra.run.smithi169.stdout: 265819 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:58 /home/ubuntu/cephtest 2024-03-25T19:58:15.082 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:58:15.094 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:58:15.173 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:58:15.201 DEBUG:teuthology.nuke:shortname: smithi069 2024-03-25T19:58:15.201 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:58:15.222 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:11.289146', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIb/u/8YPBEaSOyeWaG1lD14x3+nk7rRSq7+W3fk8olAzDvJQ2CW4ABBgm3sDGoDCeheiNDR6yrJ/Zpl13Ne74E='} 2024-03-25T19:58:15.231 DEBUG:teuthology.nuke:shortname: smithi169 2024-03-25T19:58:15.231 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:58:15.252 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621324', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:11.290883', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1vVUhWlSt6/30FLLdt0/GWS0KE8ZSYrh3OW3eK3B4lA2hSUSqcIhGCfPLa8vPgV7QJ24N99OVrmjZrrp019GA='} 2024-03-25T19:58:15.269 INFO:teuthology.orchestra.console.smithi069:Power off 2024-03-25T19:58:15.270 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:58:15.295 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-03-25T19:58:15.295 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:15.314 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-25T19:58:15.314 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:58:15.324 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-25T19:58:15.337 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-25T19:58:15.337 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:15.362 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-25T19:58:19.325 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:19.363 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:19.436 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-25T19:58:19.473 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-25T19:58:23.439 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:23.474 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:23.550 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-25T19:58:23.588 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-25T19:58:27.554 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:27.590 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:27.667 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-03-25T19:58:27.703 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-25T19:58:27.768 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-03-25T19:58:27.804 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-25T19:58:27.909 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 5574.2138113975525 failure_reason: 'valgrind error: Leak_PossiblyLost calloc __trans_list_add ' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ea0bcbf940754433adb7466407354ef4 status: fail success: false 2024-03-25T19:58:27.910 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:58:28.007 INFO:teuthology.run:FAIL